Refresh the CI image tool pins so Go-built tools are rebuilt with patched Go releases and move the sandbox Python runtime to 3.14.5.
Rebase the gateway runtime to a pinned distroless Debian 13 image with glibc 2.41-12+deb13u3 while preserving the existing UID/GID 1000 runtime identity for upgrade compatibility. Update rustls-webpki to 0.103.13 and clarify Linux k3d guidance now that k3d is not installed through mise on Linux.
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* feat(prover): add native Rust policy prover with Z3 solver
Add openshell-prover crate implementing formal policy verification
using Z3 SMT solving. Answers two questions about any sandbox policy:
"Can data leave?" and "Can the agent write despite read-only intent?"
Native Rust — no Python subprocess, no PYTHONPATH, no uv dependency.
Z3 bundled via z3-sys for self-contained builds.
Replaces the Python prototype from #703.
Closes#699
Signed-off-by: Alexander Watson <zredlined@gmail.com>
* fix(prover): skip L7-write in exfil, write bypass only on read-only intent
Port two fixes from the Python branch:
- Exfil query skips endpoints where L7 is enforced and working
- Write bypass only fires on explicit read-only intent, not L4-only
Signed-off-by: Alexander Watson <zredlined@gmail.com>
* chore(prover): add missing SPDX license headers to registry and testdata YAML files
* fix(prover): revert serde_yaml to serde_yml to match workspace dependency on main
* fix(prover): apply cargo fmt formatting to prover and cli source files
* ci: add clang and libclang-dev to CI image for z3-sys bindgen
z3-sys requires libclang at build time for bindgen to generate FFI
bindings. Without it, the Rust CI jobs fail on the prover crate.
* fix(prover): use system libz3 instead of compiling from source
Add libz3-dev to the CI image and drop the z3 `bundled` feature from
the workspace dependency. This eliminates the ~30 min z3 C++ build
that ran on every CI cache miss.
sccache only wraps rustc — it does not intercept the cmake C++ build
that z3-sys runs when `bundled` is enabled. The cargo target cache
helped on warm runs but evicts on toolchain bumps and fork PRs.
With the system library pre-installed in the CI image, z3 link time
is always zero.
A `bundled-z3` opt-in feature is added to the prover crate for local
development without system z3:
cargo build -p openshell-prover --features bundled-z3
Regular local dev: brew install z3 (macOS) or apt install libz3-dev
(Linux), then cargo build just works.
Signed-off-by: Alexander Watson <zredlined@gmail.com>
Made-with: Cursor
* fix(prover): use u16 for ports, align include_workdir default with runtime
Port fields changed from u32 to u16 across all prover types (policy,
model, finding, queries). Prevents the prover from silently accepting
port values >65535 that the runtime rejects, which would produce
misleading PASS results on invalid policies.
Change include_workdir serde default from true to false to match the
runtime (openshell-policy uses #[serde(default)] which gives false).
The previous mismatch caused the prover to model a /sandbox path that
does not exist at runtime, producing false analysis.
Signed-off-by: Alexander Watson <zredlined@gmail.com>
Made-with: Cursor
* refactor(prover): embed registry at compile time, replace hand-rolled glob
Embed binary and API capability registry YAML files into the binary at
compile time using include_dir!. The previous approach used
env!("CARGO_MANIFEST_DIR") which bakes in the build machine's source
path — works in tests but breaks for installed binaries. The CWD
fallback was equally fragile.
The --registry CLI flag still works as a filesystem override for custom
registries. Credentials remain filesystem-loaded (user-supplied data).
Replace the hand-rolled glob matching (~60 lines) with the glob crate's
Pattern::matches(), which is already a transitive dependency.
Remove unused dependencies: openshell-policy, openshell-core, thiserror,
tracing. The prover parses policy YAML directly into its own types and
does not use the policy or core crate APIs.
Signed-off-by: Alexander Watson <zredlined@gmail.com>
Made-with: Cursor
* docs: add Z3 system library to prerequisites
The prover crate now links against system libz3 instead of compiling
from source. Document the install steps for macOS, Ubuntu, and Fedora,
and note the bundled-z3 feature flag as a fallback.
Signed-off-by: Alexander Watson <zredlined@gmail.com>
Made-with: Cursor
* fix(prover): apply cargo fmt formatting
Made-with: Cursor
---------
Signed-off-by: Alexander Watson <zredlined@gmail.com>
This PR migrates the OpenShell docs to Fern and carries the follow-up cleanup needed to make the migrated site usable and close to the existing docs experience. It focuses on navigation, content structure, code block behavior, and landing-page polish, while leaving CI workflow work out of this PR.
---------
Signed-off-by: Lawrence Lane <llane@nvidia.com>
Co-authored-by: Piotr Mlocek <pmlocek@nvidia.com>
* feat(sandbox): log connection attempts that bypass proxy path
Add iptables LOG + REJECT rules inside the sandbox network namespace to
detect and diagnose direct connection attempts that bypass the HTTP
CONNECT proxy. This provides two improvements:
1. Fast-fail UX: applications get immediate ECONNREFUSED instead of a
30-second timeout when they bypass the proxy
2. Diagnostics: a /dev/kmsg monitor emits structured BYPASS_DETECT
tracing events with destination, protocol, process identity, and
actionable hints
Both TCP and UDP bypass attempts are covered (UDP catches DNS bypass).
The feature degrades gracefully if iptables or /dev/kmsg are unavailable.
Closes#268
* chore: track .python-version to pin Python 3.13.12 for uv
The sandbox base image runs Python 3.13. A stale venv on 3.12 causes
all exec_python E2E tests to fail because cloudpickle bytecode is not
compatible across minor versions.
* fix(cluster): preserve hostGatewayIP across fast deploys
The fast deploy's helm upgrade was missing the hostGatewayIP value that
the bootstrap entrypoint injects into the HelmChart CR. This caused
host.openshell.internal hostAliases to be lost from the gateway pod
and sandbox pods after any fast deploy, breaking host gateway routing.
Read the IP from the HelmChart CR and pass it through to helm upgrade.
* wip: fix iptables path resolution, use dmesg for kmsg, add CAP_SYSLOG
* fix(sandbox): restore NetworkNamespace Drop impl, remove dead kmsg code
The Drop impl for NetworkNamespace was accidentally deleted during the
bypass detection refactor, which would cause network namespaces and
veth interfaces to leak on every sandbox shutdown.
Also removes dead kmsg volume/mount code (bypass monitor uses dmesg
instead of direct /dev/kmsg access) and removes an accidentally
committed session transcript file.
* feat(policy): add policy recommendation plumbing — denial aggregation, transport, approval pipeline, and mechanistic recommendations
Implement the infrastructure layer for automated policy recommendations (#204):
- Proto: 9 new RPCs and messages for draft policy lifecycle (submit, get, approve, reject, approve-all, edit, undo, clear, history)
- Persistence: SQLite/Postgres migrations and store methods for draft_policy_chunks and denial_summaries tables
- Server: Full gRPC handler implementations with mechanistic mapper that auto-generates NetworkPolicyRule proposals from denial summaries
- Sandbox: DenialAggregator with MPSC channel, deduplication, periodic flush to gateway via SubmitPolicyAnalysis
- CLI: 'openshell draft' subcommand with get/approve/reject/approve-all/undo/clear/history operations
- TUI: Draft recommendations panel accessible from sandbox policy view
- Docs: Architecture documentation in architecture/policy-advisor.md
* feat(policy): add L7-aware mechanistic mapper and policy advisor CTF example
Add L7 rule generation to mechanistic mapper (build_l7_rules,
generalise_path, looks_like_id) with 3 new unit tests. Add
examples/policy-advisor/ with a 7-gate CTF script, restrictive
sandbox policy, and walkthrough README.
* fix(policy): use sandbox name for denial flush and add TUI draft badges
Fix denial aggregator passing sandbox UUID instead of name to
SubmitPolicyAnalysis, which caused 'sandbox not found' errors on
flush. Add notification badges to the TUI sandbox list and detail
header showing pending draft recommendation counts.
* fix(policy): deduplicate draft chunks and tolerate overlapping OPA rules
Skip draft chunk creation when a pending/approved chunk already covers
the same host:port endpoint, preventing duplicate rules across denial
aggregator flush cycles.
Rewrite three OPA complete rules (network_policy_for_request,
matched_network_policy, matched_endpoint_config) to tolerate multiple
matching policies without triggering a "complete rule conflict" error.
network_policy_for_request becomes a boolean, matched_network_policy
uses a set comprehension with min(), and matched_endpoint_config uses
an array comprehension with index-0 selection.
* feat(tui): interactive draft actions, highlight bar, and detail popup
Rework the draft recommendations panel to match the logs UX:
- Highlight bar (green accent + background) instead of arrow marker
- Viewport-aware j/k scrolling with g/G for top/bottom
- Enter opens a full-screen detail popup showing endpoints, binaries,
rationale, security notes, and action hints
Add approve/reject/approve-all draft actions:
- [a] approve selected chunk, [x] reject, [A] approve all pending
- Actions work from both the list view and the detail popup
- gRPC calls run async; result updates status bar and refreshes data
- Nav bar shows all available keybindings
Fix draft count refresh: sandbox_draft_counts now refreshes on every
tick (not just Dashboard), so the detail header badge updates in
real time.
Improve badge labels: show 'N pending' instead of a bare number in
both the dashboard sandbox list and sandbox detail header.
* refactor(policy): DB-level draft chunk dedup with hit counter and timestamps
Replace the in-memory HashSet dedup in SubmitPolicyAnalysis with a
database-level upsert. New denormalized columns on draft_policy_chunks:
- host, port: extracted from proposed_rule at insert time
- hit_count: incremented on conflict (same sandbox + host + port)
- first_seen_ms, last_seen_ms: track when the endpoint was first and
most recently proposed
A partial unique index (WHERE status IN ('pending','approved')) ensures
only one active chunk per endpoint per sandbox; rejected/superseded
chunks don't block new proposals.
Surface hit_count and first/last_seen in:
- CLI: 'openshell draft get' shows 'Hits: N (first ..., last ...)'
- TUI: detail popup shows hits row; list view shows 'Nx' suffix
* fix(policy): optimistic retry on policy version conflicts + structured logging
merge_chunk_into_policy and remove_chunk_from_policy now retry up to 5
times on UNIQUE constraint violations (version conflicts from concurrent
approvals). Each attempt re-reads the latest policy, re-merges the rule,
and increments the version. This eliminates the race condition where
rapid successive approvals would fail with a DB error.
Add structured tracing to all draft action handlers:
- ApproveDraftChunk: logs rule_name, host, port, hit_count before merge
and version + policy_hash after success
- RejectDraftChunk: logs rule_name, host, port, reason
- ApproveAllDraftChunks: logs pending_count at start, per-chunk merge
progress, and final summary with chunks_approved/skipped
- UndoDraftChunk: logs before/after with rule_name and version
- Retry attempts log as warnings with attempt number and conflicting
version
* wip: forward proxy fix, mapper allowed_ips, TUI polish, CTF rewrite
* fix(tui): use correct --gateway flag for ssh-proxy ProxyCommand
* chore: add Docker cleanup script for stale images, volumes, and build cache
* feat(tui): approve-all confirmation modal and CTF cleanup
Add [A] confirmation popup that snapshots pending chunks, shows a
scrollable list, and approves each chunk individually on confirm.
This prevents approving chunks that arrived after the modal opened.
Remove transient issue #205 reference from CTF victory banner.
* fix(tui): correct import ordering for rustfmt
* wip: stateful toggle model, rename to network rules
Draft chunks now follow a toggle state machine:
pending -> approved | rejected (initial decision)
approved <-> rejected (toggle)
One row per (sandbox_id, host, port) via expanded unique index.
Rejecting an approved rule removes it from the active policy.
Re-approving a rejected rule merges it back.
Rename CLI from 'draft' to 'rule', TUI from 'Draft Recommendations'
to 'Network Rules'. State-aware keybindings: approved shows [x] Revoke,
rejected shows [a] Approve. Fix sandbox detail hiding delete confirmation
behind pending message.
* refactor(policy): move mapper sandbox-side, slim schema, per-binary granularity
Move mechanistic mapper from gateway to sandbox so all analysis runs
sandbox-side (N sandboxes = N independent pipelines). Gateway is now a
thin validate + persist + approval layer.
Architectural changes:
- Move mechanistic_mapper.rs from navigator-server to navigator-sandbox
- Sandbox flush flow: aggregator drains -> mapper runs -> proposals sent
- Gateway SubmitPolicyAnalysis: validate + persist only, no mapper
- Drop denial_summaries table (write-only, zero readers)
- Consolidate migrations 003+004+005 into single 003
Schema slimming:
- Drop 5 unused columns from draft_policy_chunks (stage, denial_refs,
supersedes_chunk_id, analysis_mode, decided_by)
- Add per-binary granularity: binary column, widen unique index to
(sandbox_id, host, port, binary)
- Mapper groups by (host, port, binary), one proposal per triple
- Merge appends binary to existing rule; revoke removes just that binary
CTF & UX:
- 7-gate CTF: add Gate 3 (curl -> ifconfig.me:80) for per-binary demo
- TUI shows binary short name in list, full path in detail popup
- CLI output shows binary field
- Idempotent rule names, hit_count accumulates real denial counts
- Rationale text no longer bakes in stale denial count
* feat(sbom): add SBOM generation, license resolution, and CSV export tooling
Add mise-integrated SBOM pipeline for container images using Syft.
Includes license resolution via crates.io/npm/PyPI APIs and CycloneDX
JSON to CSV conversion. Adds agent skill for on-demand SBOM operations.
Closes#237
* fix(sbom): chain task dependencies to run generate → resolve → csv sequentially
* fix(sbom): add concurrent license resolution, progress logging, and exclude dev artifacts
* feat(notices): add mise run notices to generate THIRD-PARTY-NOTICES with full license texts
Use cargo-about for Rust crate licenses and pip-licenses for Python
packages. Produces a single attribution file with per-package copyright
notices and full license text for open-source compliance.
Closes#78
## What You Can Do
### Live policy updates
Update a running sandbox's network policy without recreating it:
```bash
nav sandbox policy set <sandbox> --policy new-policy.yaml --wait --timeout 60
```
The sandbox hot-reloads the policy within 30s (configurable). On failure, the previous policy stays active (last-known-good).
Idempotent — submitting the same policy twice returns the existing version:
```
✓ Policy version 3 submitted (hash: a1b2c3d4e5f6)
$ nav sandbox policy set test --policy same.yaml
· Policy unchanged (version 3, hash: a1b2c3d4e5f6)
```
### Policy history & inspection
```bash
nav sandbox policy list <sandbox> # version history with status
nav sandbox policy get <sandbox> # current policy metadata
nav sandbox policy get <sandbox> --full # print full policy as YAML
nav sandbox policy get <sandbox> --rev 2 --full # specific revision as YAML
```
### Sandbox logs
Stream logs from both the gateway and sandbox supervisor in one view:
```bash
nav sandbox logs <sandbox> # one-shot, last 2000 lines
nav sandbox logs <sandbox> --tail # live streaming
nav sandbox logs <sandbox> --source sandbox # supervisor logs only
nav sandbox logs <sandbox> --source gateway # gateway logs only
nav sandbox logs <sandbox> --level warn # warnings and errors only
nav sandbox logs <sandbox> --since 5m # last 5 minutes
nav sandbox logs <sandbox> --tail --source sandbox --level info
```
Each log line is tagged with its source and includes structured fields:
```
[1772055394.673] [sandbox] [INFO ] [navigator_sandbox::proxy] CONNECT action=allow dst_host=api.anthropic.com dst_port=443 policy=claude_code
[1772055061.005] [gateway] [INFO ] [navigator_server::grpc] GetSandboxPolicy served from policy history
```
---
## Implementation
### Proto changes
- 7 new RPCs: `UpdateSandboxPolicy`, `GetSandboxPolicyStatus`, `ListSandboxPolicies`, `ReportPolicyStatus`, `GetSandboxLogs`, `PushSandboxLogs` (client-streaming)
- `SandboxLogLine`: added `source` (gateway/sandbox), `fields` (structured key-value map)
- `WatchSandboxRequest`: added `log_since_ms`, `log_sources`, `log_min_level`
- `GetSandboxLogsRequest`: added `sources`, `min_level`
- `PolicyStatus` enum, `SandboxPolicyRevision` message
- `Sandbox.current_policy_version` field
### Server
- **Policy persistence**: New `sandbox_policies` table (SQLite + Postgres) with per-sandbox monotonic versions, status tracking, and policy hash
- **UpdateSandboxPolicy**: Validates static field immutability (filesystem/landlock/process), network mode consistency (Block↔Proxy), deterministic hash comparison for idempotent updates
- **Lazy backfill**: First `GetSandboxPolicy` call creates version 1 from `spec.policy` for existing sandboxes
- **Log broker**: `TracingLogBus::publish_external()` injects sandbox-pushed logs into the same broadcast channel + tail buffer (2000 lines). Server forces `source="sandbox"` and `sandbox_id` on all pushed logs
- **Source/level filtering**: Applied server-side in both `GetSandboxLogs` and `WatchSandbox` streams
- **Version supersession**: When a new version is loaded, all older pending+loaded versions are marked superseded
### Sandbox
- **`OpaEngine::reload_from_proto()`**: Full `from_proto()` pipeline (L7 validation, access preset expansion) with atomic engine swap. On failure, previous engine untouched (LKG)
- **Policy poll loop**: Background task polls every 30s (configurable via `NAVIGATOR_POLICY_POLL_INTERVAL_SECS`), reports status via `ReportPolicyStatus` RPC
- **`LogPushLayer`**: Tracing layer captures events at INFO+ (configurable via `NAVIGATOR_LOG_PUSH_LEVEL`), sends structured fields via `PushSandboxLogs` client-streaming RPC. Background task batches 50 lines / flushes every 500ms. Best-effort (drops on full channel, never blocks)
- **`CachedNavigatorClient`**: Persistent mTLS channel for both policy polling and log push
### Database migration
- `002_create_sandbox_policies.sql` (SQLite + Postgres)
## Tests
- **Unit**: 8 policy persistence tests (put/get/list/status/supersede/isolation)
- **Integration**: 4 test files updated with new RPC stubs
- **E2E**: `test_live_policy_update_and_logs` — full lifecycle: create → set same (unchanged) → push new → wait for load → verify connectivity → push same (unchanged) → verify history → fetch logs
## Documentation
- `architecture/sandbox.md`: Log streaming architecture, LogPushLayer, push task, server broker, source tagging, structured fields, CLI filtering, failure modes
- `architecture/security-policy.md`: Live update semantics, deterministic hashing, CLI filter flags, policy inspection
- `architecture/plans/issue-78-sandbox-log-streaming.md`: Design plan for log streaming
## Security
- Trust boundary documented: shared mTLS cert model (per-sandbox auth tracked in #80)
- Server forces `source="sandbox"` and `sandbox_id` on pushed logs (can't impersonate gateway or other sandboxes)
- Per-batch line cap (100) prevents flooding
Closes#13
## Summary
- add Python sandbox execution APIs for command and callable workflows
- consolidate sandbox policy fixtures and expand e2e test coverage for policy and Python exec paths
- update CI/build config and images for sandbox e2e execution dependencies
## Test Plan
- mise run pre-commit
John T. Myers