mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-14 04:59:28 +08:00
pull-request/2592
62
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Matthew Grossman
|
bc14018cad |
feat(sandbox): use policy-first OCI image identity (#2509)
* feat(sandbox): use policy-first OCI image identity Closes #2331 Preserve per-field policy omission, derive Docker and Podman fallbacks from the inspected immutable image, and resolve the final numeric identity before starting agent children. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(sandbox): preserve declared process identities Keep explicit policy values and OCI-declared names intact, defer passwd lookup until a primary GID is required, and refresh stale policy examples. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(supervisor): reuse resolved OCI identity Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(supervisor): allow Linux pre-exec arguments Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(kubernetes): protect resolved sandbox identity Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(sandbox): prepare workspace for OCI identity Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(sandbox): own only workspace root Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(sandbox): harden partial identity drops Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(sandbox): scope OCI image e2e to Docker Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(sandbox): narrow OCI identity fallback scope Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(podman): cover OCI identity launch Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(podman): exercise OCI fallback in E2E Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> |
||
|
Piotr Mlocek
|
2b7f04fe0a |
feat(examples): add supervisor middleware content guard (#2169)
* feat(examples): add supervisor middleware content guard Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * docs(examples): refine middleware preview warning Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(examples): add middleware policy version Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * refactor(supervisor-middleware): simplify service endpoints Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(examples): adapt content guard to middleware enums Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(examples): align content guard with merged middleware Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * feat(examples): add content guard smoke flow Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * chore(examples): remove smoke launcher test Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(examples): align content guard smoke with main Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(examples): address content guard review feedback Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * refactor(examples): parse cargo metadata with jq Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(examples): prioritize longest content matches Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * docs(examples): use GitHub warning alert Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(examples): merge overlapping content matches Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * docs(examples): render preview warning on GitHub Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> --------- Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> |
||
|
Pavel Anni
|
75d24688e1 |
fix(examples): add missing workspace fields to governance interceptor (#2436)
The ListSandboxesRequest proto gained workspace and all_workspaces fields in the workspace feature but the governance interceptor example was not updated, causing a build failure. Set all_workspaces to true so the interceptor's policy reload covers sandboxes across all workspaces. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Pavel Anni <panni@redhat.com> |
||
|
Derek Carr
|
5952a5a23f |
feat(workspace): add workspace resource model with scoping, membershi… (#2243)
* feat(workspace): implement workspace model (Phase 1 of RFC 0011) Implements workspace and membership model providing hard isolation boundaries for multi-player OpenShell deployments. Workspace CRUD with Kubernetes-style Terminating phase for graceful deletion. All resources scoped by workspace via ObjectMeta. Membership RPCs for workspace access control. Persistence migration shifts name uniqueness to (object_type, workspace, name). Provider profiles support platform and workspace scoping. Service routing uses workspace-prefixed DNS labels. Inference routes renamed and workspace-scoped with DeleteInferenceRoute RPC. Python SDK with WorkspaceClient, two-method list pattern (workspace-scoped and for_all_workspaces), and workspace parameter on all methods. CLI workspace flags, TUI workspace cycling. K8s driver filters unmanaged CRs and uses delete preconditions. Podman driver uses immutable container IDs. Label serialization fixed across all put_if call sites. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(cli): delegate sandbox upload command to existing upload function The standalone `sandbox upload` command reimplemented upload logic inline with two bugs: it used `Path::exists()` which follows symlinks (rejecting dangling symlinks), and it ran git-aware filtering on symlink sources. The `run::sandbox_upload()` function already handles both cases correctly via `sandbox_upload_plan()`. Replace the inline logic with a call to the existing function. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(e2e): shorten sandbox names and fix test compatibility Shorten the sandbox name in initial_sparse_policy_is_acknowledged_as_loaded from 'e2e-2159-sparse-enrich' (22 chars) to 'e2e-sparse-enrich' (17 chars) to comply with MAX_ROUTABLE_NAME_LEN (19 chars). Also capture stderr in create_keep_with_args so future sandbox creation failures include the actual CLI error instead of reporting empty output. Signed-off-by: Derek Carr <decarr@redhat.com> * test(workspace): add test coverage for workspace CRUD and persistence isolation Add unit tests for workspace create happy path, get round-trip, get not-found, get empty-name rejection, already-exists error, and resolve_workspace not-found. Add persistence test proving cross-workspace name uniqueness (same name in different workspaces produces separate records). Add workspace name max-length boundary tests. Fix e2e harness to include stderr in name-parse-failure error path. Align Python e2e test_workspace_crud with try/finally pattern. Document provider profile catalog workspace scoping gap in RFC 0011. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(examples): update examples for workspace model compatibility Shorten sandbox names in demo scripts to fit the 19-character MAX_ROUTABLE_NAME_LEN limit: policy-demo prefix to pd-, multi-agent notepad derives a short SANDBOX_TAG from the run ID, governance interceptor uses gs-PID-RANDOM. Update vscode-remote-sandbox.md SSH host aliases from openshell-{name} to openshell-{name}.{workspace} format. Signed-off-by: Derek Carr <decarr@redhat.com> * feat(sdk): add workspace-scoped client and workspace CRUD Add WorkspaceScopedClient modeled after kube::Api::namespaced — captures workspace once and injects it into every sandbox request. Add workspace CRUD methods (create, get, list, delete) and list_sandboxes_all_workspaces on OpenShellClient. Extend SandboxRef with workspace field and add WorkspaceRef type. Include mock tests for all new operations. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(lint): resolve clippy warnings in workspace test assertions Signed-off-by: Derek Carr <decarr@redhat.com> * fix(docs): convert indented code blocks to fenced in RFC 0011 Signed-off-by: Derek Carr <decarr@redhat.com> * fix(lint): resolve clippy warnings and apply cargo fmt across workspace Auto-format with cargo fmt and fix clippy warnings exposed by the reformat: unnecessary qualifications, map_unwrap_or, identical match arms, unused variable prefix, dead code annotations, and let-unit-value in e2e harness. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(workspace): address workspace scoping issues from review - Add workspace field to settings JSON output (CLI) - Skip Podman containers missing workspace label instead of defaulting to empty string, matching K8s driver behavior - Add resource_version to list_by_scope SELECT in both SQLite and Postgres backends, with regression test - Gate PolicyLocalContext proposal/lookup routes on workspace readiness, returning 503 when workspace is not yet discovered - Block sandbox and provider creation in TUI all-workspaces mode - Clear workspace vectors in TUI reset_sandbox_state Signed-off-by: Derek Carr <decarr@redhat.com> * fix(workspace): make provider profile catalog workspace-aware Thread workspace through snapshot_catalog so the EffectiveProviderProfileCatalog enforces workspace boundaries on both read and write paths. UserProviderProfileSource now loads platform-scoped profiles (workspace "") plus the target workspace's profiles, preventing cross-workspace duplicate profile ID collisions that previously caused global catalog failures. Update RFC 0011 to reflect catalog scoping is implemented in Phase 1 rather than deferred to future work. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(persistence): include workspace column in atomic policy revision INSERT put_policy_revision_atomic omitted the workspace column from the INSERT into the objects table in both SQLite and Postgres backends, causing atomically-written policy revisions to lose their workspace association. Add workspace field to AtomicPolicyRevisionWrite and thread it through both backend INSERT statements, matching the non-atomic put_policy_revision path which already included it. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(proxy): skip ancestor walk when socket owner is the entrypoint collect_ancestor_identities walked the entire process tree above the entrypoint when the connecting process was the entrypoint itself, SHA256-hashing every ancestor binary (IDE, shell, container runtime). On dev machines with large binaries in the ancestor chain this exceeded the 30-second test timeout. When start_pid == stop_pid there are no intermediate ancestors to verify, so return an empty list immediately. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(workspace): make provider profile catalog scope-aware Allow the same profile ID at platform and workspace scopes by introducing layered catalog entries where workspace profiles shadow platform profiles. Add source and scope fields to the ProviderProfile proto and CLI output. Migrate List/Get handlers to the catalog, fixing divergence with runtime profile resolution. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(e2e): align podman e2e labels with centralized driver constants The podman driver moved its container labels to the centralized openshell.ai/ prefix, but the e2e test harness and cleanup script still referenced the old openshell.sandbox-* keys, causing the local_driver_token_restart test to fail on container lookup. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(e2e): align python profile isolation test with scope-aware catalog Platform profiles are now visible in workspace listings as fallbacks per the layered catalog design. Update the assertion to match. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(workspace): honor profile_workspace in runtime profile resolution Runtime profile lookups now consult provider.profile_workspace via get_type_profile_for_scope. Providers created with --global-profile (profile_workspace="") resolve to the platform profile even when a workspace profile shadows the same ID. All 6 runtime call sites updated; type-only call sites remain scope-agnostic. Signed-off-by: Derek Carr <decarr@redhat.com> --------- Signed-off-by: Derek Carr <decarr@redhat.com> |
||
|
Mesut Oezdil
|
8cf2673c0b |
docs: bump stated Rust MSRV from 1.88 to 1.90 (#2276)
* docs: bump stated Rust MSRV from 1.88 to 1.90 Cargo.toml sets rust-version = "1.90" (rust-toolchain.toml pins 1.95.0), so building with the previously documented 1.88 fails Cargo's MSRV check. * docs: bump e2e/rust MSRV to 1.90 * fix: align remaining Rust version fields to 1.90 examples/governance-interceptor/Cargo.toml still had rust-version 1.88. Also bump e2e/rust's prost dependency to 0.14 to match the workspace, since it was on 0.13 in an otherwise standalone crate. |
||
|
Russell Bryant
|
aa483ecb9a |
feat(providers): AWS STS AssumeRole refresh strategy and aws-s3 profile (#1782)
Add gateway-managed AWS STS credential refresh (provider-v2, #1576). The gateway calls sts:AssumeRole and writes three short-lived credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN) to the provider record; the proxy re-signs requests with SigV4. Adds the aws and aws-s3 provider profiles and a declarative multi-output refresh model (additional_outputs) so one AssumeRole co-mints all three credentials. Signed-off-by: Russell Bryant <rbryant@redhat.com> |
||
|
Drew Newberry
|
83003e80fc |
feat(interceptors): initial gateway interceptor implementation and reference example (#2005)
* feat(gateway): add descriptor-driven interceptors Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(gateway): add service-reflected interceptors Signed-off-by: Drew Newberry <anewberry@nvidia.com> * wip * fix(gateway): harden interceptor evaluation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(interceptors): label metrics and harden governance smoke Signed-off-by: Drew Newberry <anewberry@nvidia.com> * remove on_error: ignore * feat(gateway-interceptors): emit log annotations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(examples): govern provider profiles in interceptor Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(gateway): preserve update config annotations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(providers): support interceptor profile catalogs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * wip * feat(governance-interceptor): sign provider profiles Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(providers): use configured profile sources for refresh updates Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(gateway-interceptors): add phase-specific evaluation payloads Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(providers): compose provider profile sources Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(gateway-interceptors): preserve committed responses Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(gateway-interceptors): reject ambiguous protobuf oneofs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(gateway-interceptors): validate patch candidates per binding Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(gateway-interceptors): use reflected protobuf codec Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(governance-example): canonicalize signed protobuf hashes Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(gateway): commit policy provenance atomically Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(gateway): close signed governance bypasses Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(gateway): isolate interceptor secrets and authority Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(gateway): snapshot provider profiles per request Signed-off-by: Drew Newberry <anewberry@nvidia.com> * chore(gateway): resolve server clippy warnings Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(server): satisfy provider source clippy lint Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(server): initialize policy test annotations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(gateway-interceptors): require explicit route allowlist Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(proto): clarify update annotation semantics Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Ignas Baranauskas
|
a72711697d |
chore: remove deprecated --keep flag from docs, scripts, and e2e tests (#2126)
* docs: remove deprecated --keep flag from tutorials and examples The --keep flag is deprecated, hidden, and a no-op since sandboxes are kept by default. Remove references from tutorial docs and example READMEs that explain it as a real feature. - Remove --keep from sandbox create commands - Remove --keep explanation text - Clarify that sandboxes are kept by default Signed-off-by: Ignas Baranauskas <ibaranau@redhat.com> * chore: remove deprecated --keep usage from scripts and e2e tests The --keep flag is a deprecated no-op since sandboxes are kept by default. Stop passing it in internal scripts, e2e test scripts, and example demo scripts. Signed-off-by: Ignas Baranauskas <ibaranau@redhat.com> --------- Signed-off-by: Ignas Baranauskas <ibaranau@redhat.com> |
||
|
Mesut Oezdil
|
9c14de7b83 | docs: fix article before OpenShell in sync-files (#2133) | ||
|
Seth Jennings
|
6461677c32 |
feat(policy): accept numeric UIDs for sandbox process identity (#1973)
* feat(policy): accept numeric UIDs in sandbox process identity validation Allow run_as_user and run_as_group to be either the literal 'sandbox' or a numeric UID/GID within [1000, 2_000_000_000]. This removes the hard dependency on a baked-in 'sandbox' user in container images, enabling compute drivers to inject resolved UIDs at sandbox creation. Phase 1 of #1959. Signed-off-by: Seth Jennings <sjenning@redhat.com> * feat(supervisor): accept numeric UIDs for process identity dropping Allow run_as_user and run_as_group to be numeric UIDs/GIDs, removing the hard dependency on a baked-in 'sandbox' user in container images. Changes: - validate_sandbox_user(): accepts numeric UIDs without passwd lookup (logs OCSF event); keeps passwd check for "sandbox" name; rejects non-numeric non-sandbox strings that fail passwd lookup - prepare_filesystem(): passes numeric UIDs/GIDs directly to chown() instead of requiring a passwd entry - drop_privileges(): resolves numeric UIDs/GIDs directly via UID::from_raw / Gid::from_raw; skips initgroups when target uid matches current euid; uses guard conditions before setgid/setuid calls - session_user_and_home(): falls back to ("{uid}", "/sandbox") for numeric UIDs, avoiding a passwd lookup that will fail Re-exports MIN_SANDBOX_UID and MAX_SANDBOX_UID from openshell-policy so callers have consistent range constants. Phase 2 of #1959. Signed-off-by: Seth Jennings <sjenning@redhat.com> * feat(driver-kubernetes): resolve sandbox UID/GID from config or OpenShift SCC annotations Phase 3 of the numeric-UID plan: allow operators to specify explicit sandbox_uid/sandbox_gid in Kubernetes driver config, auto-detect from OpenShift SCC namespace annotations, and propagate resolved values to supervisor container env vars and PVC init container securityContext. Changes: - Add sandbox_uid/sandbox_gid fields to KubernetesComputeConfig - Add SANDBOX_UID/SANDBOX_GID env var constants to openshell-core - Implement resolve_sandbox_identity() to fetch namespace annotations and auto-detect OpenShift SCC UID ranges (sa.scc.uid-range) - Pass resolved UID/GID through SandboxPodParams to pod spec builder - Inject SANDBOX_UID/SANDBOX_GID env vars into supervisor container - Update PVC init container securityContext with resolved UID/GID instead of hard-coded root - Add comprehensive unit tests for resolution logic and annotation parsing (resolve_sandbox_uid, resolve_sandbox_gid, OpenShift SCC annotation parsing) Signed-off-by: Seth Jennings <sjenning@redhat.com> * feat(driver-vm): add configurable sandbox UID/GID and update docs/examples Phase 4 of the numeric-UID plan: replace hardcoded SANDBOX_UID (10001) in VM rootfs preparation with configurable sandbox_uid/sandbox_gid fields. Changes: - Add sandbox_uid/sandbox_gid to VmDriverConfig with serde derives - Pass resolved UID/GID through prepare_sandbox_rootfs_from_image_root to ensure_sandbox_guest_user which writes /etc/passwd/group/gshadow - Update BYOC Dockerfile: remove groupadd/useradd, document runtime UID injection and the ability to skip baked-in sandbox user - Update gateway-config.mdx: document sandbox_uid/sandbox_gid for both Kubernetes (with OpenShift SCC autodetection) and VM drivers - Update sandbox-compute-drivers.mdx: add Sandbox User Identity section explaining numeric UID support across all compute drivers - Update rootfs tests to use non-default UIDs, verify config passthrough Signed-off-by: Seth Jennings <sjenning@redhat.com> * code review changes * fix(supervisor): harden tests for restricted CI container environments Guard tests against CI-specific constraints: root without CAP_SETPCAP, UIDs with no /etc/passwd entry, and restricted /proc access. Signed-off-by: Seth Jennings <sjennings@nvidia.com> Signed-off-by: Seth Jennings <sjenning@redhat.com> --------- Signed-off-by: Seth Jennings <sjenning@redhat.com> Signed-off-by: Seth Jennings <sjennings@nvidia.com> |
||
|
Taylor Mutch
|
702cbc4f63 |
feat(providers): support SPIFFE-backed token grants (#1784)
* feat(providers): support SPIFFE-backed token grants Add provider profile token_grant metadata and expand endpoint-specific dynamic credentials so sandbox supervisors can request SPIFFE JWT-SVIDs, exchange them with an OAuth-style token endpoint, cache returned access tokens, and inject bearer tokens into matching HTTP requests. Wire Kubernetes and Helm deployments to mount the provider SPIFFE Workload API socket into sandbox pods for token grant exchange. Signed-off-by: Taylor Mutch <taylormutch@gmail.com> Signed-off-by: Gordon Sim <gsim@redhat.com> * test(examples): add SPIFFE token grant demo Add a reusable alpha/beta demo that deploys a SPIFFE-verifying token issuer and protected services, imports a token-grant provider profile, creates a sandbox, and verifies endpoint-specific bearer tokens. The script leaves Kubernetes workloads in place, deletes sandboxes through openshell unless KEEP_SANDBOX=1, and prints protected service logs as proof of life. Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(providers): harden SPIFFE token grants * fix(providers): harden dynamic token grants Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(providers): harden token grant handling --------- Signed-off-by: Taylor Mutch <taylormutch@gmail.com> Signed-off-by: Gordon Sim <gsim@redhat.com> |
||
|
Mesut Oezdil
|
8bf667f377 | fix: update RFC link in agent-driven-policy-management README (#1677) | ||
|
Mesut Oezdil
|
ae5127f14d |
fix: correct example paths in local-inference README (#1676)
* fix: correct example paths in local-inference README * fix: correct example paths in local-inference routes.yaml |
||
|
Alexander Watson
|
e98ea3ee93 | feat(policy): add agentic approval loop (#1528) | ||
|
Drew Newberry
|
603b3e27fa | docs: update NemoClaw/OpenClaw references (#1529) | ||
|
Russell Bryant
|
c5d1d76d94 |
refactor(sandbox): replace iptables with nftables for network policy enforcement (#1401)
Migrate all sandbox and VM driver network policy enforcement from iptables to nftables. nftables provides atomic ruleset loading, a cleaner rule syntax, and is the standard netfilter interface in modern kernels. Sandbox bypass enforcement (openshell-sandbox): - Replace iptables chain of individual rule insertions with a single atomic nftables ruleset load via nft -f - New nft_ruleset module with pure functions for ruleset generation and unit tests - Combine log and reject rules in one inet family table (handles both IPv4 and IPv6 in a single ruleset) - Fall back to reject-only ruleset when kernel lacks nft_log support - Enable net.netfilter.nf_log_all_netns so log rules work from non-init network namespaces - Use temp file for nft ruleset loading instead of stdin for compatibility with minimal VM guest environments VM TAP networking (openshell-driver-vm): - Replace iptables NAT/forwarding rules with nftables equivalents - New nft_ruleset module for TAP network rule generation with unit tests - Atomic table-per-TAP-device lifecycle (create/destroy) - Host-side rules provide NAT infrastructure and defense-in-depth isolation (input chain restricts VM to gateway port only, forward chain blocks unsolicited inbound); primary security enforcement happens inside the VM guest via the sandbox supervisor's own rules VM init script: - Load nft kernel modules at sandbox init - Enable nf_log_all_netns sysctl for bypass detection logging OCSF / docs: - Update firewall rule engine references from iptables to nftables - Document host firewall interaction model and two-layer enforcement architecture in VM driver README and compute drivers reference Closes #1335 Signed-off-by: Russell Bryant <rbryant@redhat.com> |
||
|
Mesut Oezdil
|
1c317646c6 |
docs: replace --sync with --upload . in sync-files example (#1366)
The --sync flag was never implemented. The correct flag is --upload with a path argument. Updated examples and workflow snippet to reflect the actual CLI interface. Fixes #1339 Signed-off-by: Mesut Oezdil <versusfinem@gmail.com> |
||
|
Alexander Watson
|
ea2fddbe2d |
feat(policy): agent-driven policy management — the agent half (#1323)
* feat(policy): plumb chunk_ids and rejection_reason through proposal pipeline Prereq plumbing for the agent revise-and-resubmit loop. Two narrow additive proto changes unblock the upcoming /wait endpoint (#1092), prover validation badge (#1097), and reject --guidance surfaces (#1098). - SubmitPolicyAnalysisResponse: add accepted_chunk_ids so the in-sandbox agent gets handles to watch its proposals. Surfaced through the typed grpc_client wrapper and policy.local's POST /v1/proposals 202 body. Closes #1094. - PolicyChunk + StoredDraftChunk + DraftChunkPayload: add validation_result (gateway prover verdict, populated by #1097) and rejection_reason (operator free-form text). Both plain strings; no enums, no parsing on the read path. Closes #1096. - RejectDraftChunk now persists the existing reason field into the chunk's rejection_reason so it round-trips back to the agent via GetDraftPolicy. UndoDraftChunk clears it on the way back to pending so consumers cannot read a stale guidance string from a prior reject -> re-approve -> undo cycle. Whole surface stays gated behind agent_policy_proposals_enabled. Two focused tests cover the round-trip and the undo-clears guarantee. Signed-off-by: Alexander Watson <zredlined@gmail.com> * feat(sandbox): add /v1/proposals/{id} and /wait long-poll to policy.local The agent feedback channel back from policy.local. Two new routes let the in-sandbox agent learn its proposal's outcome on a single blocking HTTP call — zero LLM tokens during the wait. - GET /v1/proposals/{chunk_id} returns the chunk's current state in one gateway call. - GET /v1/proposals/{chunk_id}/wait?timeout=<s> blocks until the chunk transitions out of pending. Default 60s, clamped [1, 300]. Agent re-issues on timeout to extend. Response carries the chunk's status plus the two feedback fields shipped in the prereq commit: rejection_reason (free-form reviewer text) and validation_result (gateway prover verdict, empty until #1097). On timeout: same shape with timed_out: true so the agent can disambiguate without parsing. Wait handler short-polls GetDraftPolicy every 1s inside the request with a tokio::time::Instant deadline. One gateway connection is opened per request and reused across all polls, so a 60s wait does one TLS handshake instead of sixty. A future commit can swap the loop body for a tokio::sync::broadcast driven by a watcher task — the agent-visible contract (URL, query, response shape) is independent of the polling implementation. All routes stay behind agent_policy_proposals_enabled. Closes #1092. Signed-off-by: Alexander Watson <zredlined@gmail.com> * docs(sandbox): teach policy_advisor skill the wait + redraft loop The agent-facing instructions for the feedback loop. The endpoints exist; this is the doc that makes them usable. policy_advisor.md gains: - API entries for GET /v1/proposals/{chunk_id} and /wait?timeout=<s>, including the field semantics (status, rejection_reason, validation_result, timed_out). - A note on the submit response's accepted_chunk_ids / rejection_reasons split so the agent handles partial acceptance. - Step 6 saves the chunk_ids and addresses any submit-time rejections before waiting. - Step 7 walks the four wait outcomes: approved (retry, with the honest "may still fail" caveat), rejected (read rejection_reason AND validation_result; address whichever has content), still-pending with timed_out (re-call), non-2xx (surface, do not retry). skills.rs gains two assertions on the skill content so a future edit cannot drop the wait endpoint or the rejection_reason directive silently. Closes #1095. Signed-off-by: Alexander Watson <zredlined@gmail.com> * test(policy-advisor): add end-to-end smoke for the agent feedback loop A focused smoke that exercises the new policy.local /wait endpoint on a live gateway + sandbox, separate from the existing no-LLM regression harness (which still drives the OLD retry-with-bash-loop recovery pattern). Two flows: - Flow A — approve-and-retry: agent submits, /wait blocks, host runs `openshell rule approve`, /wait returns status=approved. Confirms the happy path round-trip latency. - Flow B — reject-with-guidance: agent submits, /wait blocks, host runs `openshell rule reject --reason "..."`, /wait returns status=rejected with the exact reviewer text in rejection_reason. Confirms the free-form guidance contract round-trips through the agent feedback channel. No GitHub credentials needed — proposals are synthetic and never trigger outbound traffic. Both flows expect agent_policy_proposals_enabled=true and a running gateway. Adds three cases to sandbox-runner.sh: submit-test-proposal (no GH deps), proposal-status, proposal-wait. The existing put-file and submit-proposal cases used by test.sh are untouched. Signed-off-by: Alexander Watson <zredlined@gmail.com> * fix(policy-advisor): surface real CLI errors from wait-smoke preflight The preflight piped openshell's stderr to /dev/null and relied on jq to default the missing setting key to "<unset>", but under `set -euo pipefail` a non-zero exit from openshell makes the whole pipeline fail and the command substitution exits the script silently before the intended fail() message can print. Capture stderr explicitly, check the CLI exit code, and surface the real error plus the expected fix (port-forward + gateway add + select) when the CLI cannot reach the gateway. Signed-off-by: Alexander Watson <zredlined@gmail.com> * fix(policy-advisor): pass --json to settings get in wait-smoke preflight `openshell settings get --global` defaults to a human-readable table; jq cannot parse it and the preflight died with a numeric-literal error. Pass --json so jq gets actual JSON. Also touched up the suggested recovery commands in the preflight error to match the real CLI shape (`gateway add <endpoint> --name <name>` and the env-var override warning). Signed-off-by: Alexander Watson <zredlined@gmail.com> * fix(policy): dedup draft chunks only in mechanistic mode; return effective id The smoke harness for the agent feedback loop caught a real bug in the gateway: SubmitPolicyAnalysis's response carried a chunk_id that was never persisted whenever the SQL ON CONFLICT path fired. Two failure modes, both load-bearing: - Agent-authored proposals targeting the same host/port/binary (e.g. the redraft-after-rejection loop) silently folded into one row and any RejectDraftChunk by the new chunk_id failed with "chunk not found." Latent since #1151, surfaced by #1094 returning chunk_ids. - Mechanistic mode had the same class of bug — the dedup fold-in is the intended behavior there, but the response still advertised the newly-generated UUID instead of the existing row's id. Less visible because no current caller reads mechanistic chunk_ids back, but the proto contract was violated either way. Fix in three parts: - put_draft_chunk now takes Option<&str> dedup_key explicitly and returns the effective row id (via RETURNING). None binds NULL to the dedup_key column, which bypasses the partial-index ON CONFLICT path entirely. Caller-decides semantics replace store-side magic. - handle_submit_policy_analysis picks dedup_key per chunk using an allowlist (only "mechanistic" dedups) and pushes the returned effective_id to accepted_chunk_ids. New modes default to no-dedup so a misconfigured caller cannot silently lose proposals. - The two-copy draft_chunk_dedup_key helper consolidated to one observation_dedup_key in policy_store.rs with a doc comment. Tests: - agent_authored_submits_for_same_endpoint_do_not_dedup pins the redraft-loop contract: two intentional submissions with the same host/port/binary get distinct chunk_ids, both findable via GetDraftPolicy, both rejectable by id. - mechanistic_submits_for_same_endpoint_dedup_into_one_chunk locks in the observation-mode dedup AND asserts both submits return the same effective_id — would have caught the deeper bug. Proto: SubmitPolicyAnalysisRequest.analysis_mode doc updated to describe the actual semantics (mechanistic dedups, agent_authored and unknown modes do not). Signed-off-by: Alexander Watson <zredlined@gmail.com> * docs(examples): retarget policy-management demo at the /wait endpoint The narrated demo (examples/agent-driven-policy-management) has been the public face of this feature since #1151. Its agent prompt told Codex to retry the original PUT every few seconds for up to 120 seconds — a polling workaround for the missing /wait endpoint that this branch shipped. Update the demo to exercise /wait so the canonical reading of the feature reflects the actual UX win. - agent-task.md: step 4 is now "call /wait, branch on status" with the three outcomes spelled out (approved → retry once; rejected → read rejection_reason and revise or stop; pending+timed_out → re-issue /wait once, do NOT busy-loop or shorten the timeout). Also makes explicit that the demo submits one rule per proposal so accepted_chunk_ids[0] is the safe single id to wait on. - demo.sh: header docstring rewritten as a six-step loop that mirrors the README. narrate_sandbox_workflow drops its parallel numbering and uses bullets (the runtime narration is the agent's sub-actions, not a separate decomposition of the loop). Approve step header and success message now reference /wait waking the agent, not "policy hot-reload retry." - README.md: top-of-file flow expanded from 5 to 6 steps to include the /wait call and chunk_id capture; "Going further" section now describes both regression scripts and the boundary between them (real-GitHub retry vs. synthetic /wait wire test). Slow-path qualifier corrected from "image pull on first run" to "sandbox cold-start (SSH bring-up plus Codex install)". - wait-smoke.sh header rewritten to make it unambiguous this is a regression, NOT a tutorial, with explicit prereq commands instead of prereq descriptions, and a pointer at demo.sh for the narrated story. No code paths change; this is the readability pass. Signed-off-by: Alexander Watson <zredlined@gmail.com> * fix(examples): pass --yes on demo.sh's global setting writes Global setting updates require explicit confirmation in non-interactive mode; demo.sh's enable_agent_proposals and the cleanup restore path were missing --yes and hard-failed the preflight. Pre-existing issue that surfaced now that more of the demo runs through this path. No other behavior change. Signed-off-by: Alexander Watson <zredlined@gmail.com> * feat(sandbox): emit OCSF audit events for policy proposal lifecycle The demo's policy decision trace previously showed only the proxy enforcement story (HTTP:PUT DENIED, CONFIG:LOADED, HTTP:PUT ALLOWED). It was silent about who proposed what or who decided what — the audit-trail receipts for the agent feedback loop were missing. policy.local now emits sandbox-side OCSF events at the observation moments, into the same stream as the existing CONFIG:LOADED: - CONFIG:PROPOSED on submit_proposal acceptance. Per accepted chunk: the message names the chunk_id, target endpoint, L7 method/path, and binary so the trace correlates against the inbox card via chunk_id. - CONFIG:APPROVED on /wait observation of approved status. - CONFIG:REJECTED on /wait observation of rejected status. Carries the reviewer's free-form rejection_reason in the message AND as an unmapped field, both sanitized (control chars stripped, capped at 200 chars with an ellipsis marker). The agent still reads the raw text via GET /v1/proposals/{id}; sanitization is audit-side only, per AGENTS.md's no-secrets-in-OCSF rule. The submit path defends the audit_summaries / accepted_chunk_ids index pairing against a future gateway change that compresses past rejected chunks (the proto doesn't promise 1:1 ordering with the request). Today client-side validation makes the lengths always match; if they don't, the pairing falls back to a generic per-id event rather than mis-attribute. The wait handler's emit site fires once per terminal-status observation. Multiple concurrent waiters on the same chunk would each emit one event; acceptable for single-waiter-per-chunk demos and the right place to dedup is the SIEM. demo.sh's trace filter now surfaces the four CONFIG: events alongside HTTP:PUT, so the trace at the end of every run tells the full story from deny to allow via propose -> approve. wait-smoke.sh's prereq notes recommend redirecting kubectl port-forward output so its "Handling connection for 8090" lines don't bleed into demo narration. Three new unit tests on the sandbox-side helpers — summary builder happy path, fallback, and the rejection_reason sanitizer. Signed-off-by: Alexander Watson <zredlined@gmail.com> * feat(policy): /wait awaits local policy reload; demo auto-approves redrafts Three things in one commit, all surfaced by running the demo end-to-end against a real gateway and finding the agent had to draft a broader second proposal. 1. /wait race fix. Previously /wait returned `approved` the moment it observed the gateway's chunk status flip, but the local supervisor reloads policy on its own poll cycle (~10s in practice). The agent's retry would race the reload and hit the still-old policy, getting denied. Codex then drafted a broader rule and re-submitted — sound agent behavior, but not what /wait should provoke. Now /wait captures the local policy version at start, and after observed-approved waits for the supervisor to load a strictly-newer version before returning. Bounded by the caller's deadline; best-effort return if the deadline elapses without the version bumping. Two new unit tests pin the happy path and the deadline-clamped fallback. 2. demo.sh auto-approve loop. Replaces approve_when_pending + wait_for_agent with one approve_pending_until_agent_exits function that keeps watching for pending chunks and approving them until the agent process exits (or the configured timeout). Defense in depth against future redraft scenarios for any reason; today (post-fix #1) the agent should only submit one proposal per task, but we don't want to hang silently if it does submit more. 3. UX. Step headers now carry "[t+1.2s]" relative timestamps so reading the run output makes latency visible (the demo's whole point is the wait is cheap — surface that). A spin_wait helper renders an ASCII spinner during the watch loop so the demo never looks frozen on a TTY. Falls back to plain sleep on non-TTY contexts. Closes the race condition diagnosed from the trace timing where the gateway approved at t+0, sandbox observed at t+0.3s, but the supervisor didn't load v2 until t+9.4s — well after the agent had already retried and been denied. Signed-off-by: Alexander Watson <zredlined@gmail.com> * fix(sandbox): /wait detects policy reload by content, not the schema version The previous attempt at the /wait-after-approve race fix compared `SandboxPolicy.version` between /wait start and the policy reload — but that field is the *schema* version (constant 1), not a revision counter. Every comparison was `current(1) > baseline(1) == false`, so the wait blocked until the agent's 300s timeout regardless of whether the supervisor had actually reloaded. The demo SSH connection then timed out around the 240s mark. Diagnosed from a live run's OCSF trace: supervisor pulled v2 at +8.5s after approval (CONFIG:LOADED), but the sandbox-side CONFIG:APPROVED that my /wait emits didn't fire until +304s — exactly at the 300s deadline. Fix: compare the whole policy via prost's derived PartialEq. Any field change (network_policies map being the only one that actually mutates today) flips equality. A clone-per-200ms-tick on a few-KB proto is cheap inside the bounded wait window. Tests rewritten to match the new contract: the supervisor-reload fixture now keeps `version: 1` constant and changes `network_policies` contents, mirroring the exact failure mode from the live run. Signed-off-by: Alexander Watson <zredlined@gmail.com> * fix(examples): redact tokens with python literal-string replace, not sed The sed-based redact_log in demo.sh broke when one of the auth tokens contained a character that conflicted with sed's pattern parser ("unterminated substitute pattern" on the Codex JWT). The whole log tail then blanks on failure, hiding the very failure context we're trying to surface. Switch to a python subprocess that takes the tokens via argv and does literal str.replace. No regex, no delimiter games, no truncation. Signed-off-by: Alexander Watson <zredlined@gmail.com> * fix(sandbox): scope /wait reload check to the approved rule Reviewer (John Myers) flagged two failure modes in the prior whole-policy fingerprint approach used by policy.local /wait: - False sleep: when the supervisor reloads between two /wait calls (the skill tells the agent to re-issue on timed_out), the new call snapshots the already-updated policy as baseline and burns the full timeout waiting for a change that never comes. - False wakeup: any unrelated reload (other agent's approval, settings change) flips the diff, but the chunk's actual rule may not be loaded yet — the agent retries and hits policy_denied for no real signal. Replace the diff with rule-coverage. New public helper openshell_policy::policy_covers_rule reuses endpoints_overlap (so it matches add_rule's merge semantics, including the fold-into-existing-key case) plus an L7 allow check on method/path (so an existing endpoint that doesn't yet contain the proposed method doesn't signal coverage). Add policy_reloaded: true|false to the /wait response on approve, with a 500ms floor on the reload-wait phase so approvals arriving near the deadline still get a fair shot at reloaded=true. Update the policy_advisor skill to branch on it: reloaded=true → retry; reloaded=false → re-issue /wait once with timeout=30, then surface to user. Don't loop tightly. Tests: - 9 new unit tests in openshell-policy pinning coverage semantics (L4-only, L7 method gap, fold-into-existing-key, empty binaries). - 4 new tokio tests in policy_local mirroring John's exact scenarios. - wait-smoke.sh asserts policy_reloaded=true on Flow A. Signed-off-by: Alexander Watson <zredlined@gmail.com> * fix(server): make GetDraftPolicy dual-auth so /wait works under OIDC policy.local calls GetDraftPolicy from inside the sandbox supervisor via the sandbox gRPC client, which authenticates with the shared x-sandbox-secret. GetDraftPolicy was listed only in the Bearer-auth scope table (config:read) and was not in SANDBOX_SECRET_METHODS or DUAL_AUTH_METHODS, so OIDC-enabled gateways rejected those calls and the /wait long-poll surfaced gateway_lookup_failed. Local/no-OIDC setups happened to work because the auth check is short-circuited. Add GetDraftPolicy to DUAL_AUTH_METHODS, matching the existing GetSandboxConfig pattern (called by both CLI reviewer surfaces with Bearer and the sandbox supervisor with x-sandbox-secret). Dual-auth short-circuits the scope check for sandbox-secret callers, so the config:read entry in authz.rs continues to gate Bearer-only flows. Mirror the openshell_get_sandbox_config_is_dual_auth assertion for GetDraftPolicy. Note: ssh_handshake_secret is server-wide, not per-sandbox, so a sandbox-secret caller can today name any sandbox in a SubmitPolicyAnalysis request — and now in a GetDraftPolicy request. The exposure is symmetric with the existing SANDBOX_SECRET_METHODS pattern. Filed as a follow-up: per-sandbox secret binding, tracked separately. Signed-off-by: Alexander Watson <zredlined@gmail.com> * fix(ci): address rebased check failures Signed-off-by: Alexander Watson <zredlined@gmail.com> --------- Signed-off-by: Alexander Watson <zredlined@gmail.com> |
||
|
Alexander WatsonandJohn Myers
|
1c79b2131f |
feat: agent-driven policy management MVP (#1151)
* docs(rfc): add agent-driven policy management * docs(rfc): switch policy MVP to local API * docs(rfc): clarify policy advisor skill and local logs * feat(sandbox): add agent-driven policy proposal loop * test(examples): add codex policy dogfood loop * refactor(examples): make policy demo agent-agnostic * refactor(examples): colocate policy validation harness * docs(examples): add policy demo env sample * docs(examples): use placeholder env example * feat(sandbox): wire policy.local denials to OCSF JSONL log Wires GET /v1/denials?last=N on the sandbox-local policy advisor API to read recent OCSF JSONL events from /var/log/openshell-ocsf.YYYY-MM-DD.log, filter to network/L7 denials (action_id=2, class_uid 4001/4002), and return a compact summary newest-first. Default limit is 10, capped at 100. Ran inside spawn_blocking so file I/O does not block the policy.local handler. Other cleanup: - POST /v1/proposals now uses the typed grpc_client wrapper instead of raw_client, so accepted/rejected counts surface to the agent uniformly. Wrapper return type extended to the response struct. - Drop the 'add_rule' snake_case alias in the proposal JSON; canonical form is camelCase 'addRule', matching the PolicyMergeOperation convention used elsewhere. - skills/policy_advisor.md updated to match: documents the now-real /v1/denials?last=10 endpoint and uses 'addRule' consistently. - skills.rs test asserts on the canonical 'addRule' phrase rather than the removed 'PolicyMergeOperation' substring. * feat(cli): show L7 protocol/method/path in rule get output format_endpoint() previously rendered only host:port, dropping protocol, access, and the L7 rules array. That made openshell rule get text output unable to distinguish a broad L4 grant from a method/path-scoped L7 REST rule -- exactly the distinction a developer needs at approval time. New rendering tags each endpoint with its enforcement layer and surfaces allow/deny rules: bare L4: api.example:443 [L4] L7 read-only: api.example:443 [L7 rest, access=read-only] L7 method/path: api.example:443 [L7 rest, allow PUT /v1/foo/bar] Pure display change: no proto, gateway, or behavior changes. Unit test covers all three rendering cases with synthetic fixtures. * refactor(examples): rewrite policy demo as Codex-default loop Re-shape examples/agent-driven-policy-management/ to be a single, clean end-to-end demonstration of the agent-driven policy loop. A Codex agent inside an OpenShell sandbox attempts a GitHub Contents API write, hits a structured 403 from the L7 proxy, reads the policy_advisor skill, drafts a narrow addRule proposal via http://policy.local/v1/proposals, the host auto-approves, the sandbox hot-reloads policy, and the agent's retry succeeds. Whole loop runs in roughly two minutes. Demo cleanup: - Drop .env file ceremony. Defaults resolve from gh: owner via 'gh api user --jq .login', repo defaults to 'openshell-policy-demo', token from gh auth token / GITHUB_TOKEN / GH_TOKEN. With gh auth login and codex login already done, 'bash demo.sh' Just Works. - Codex-specific. Bootstraps ~/.codex/auth.json from credentials injected by the OpenShell provider, runs codex exec --sandbox danger-full-access (OpenShell is the actual security boundary; bwrap nesting cannot create user namespaces inside the sandbox container). - Tighter narrative output: a single 'Preflight' step, a run summary banner before launch, an inline narration of what's happening inside the sandbox while we poll for the proposal (including the literal structured 403 body the agent acts on), and an OCSF trace at the end filtered to the three events that tell the story (DENY, RELOAD, ALLOW). - Replace Python heredoc templating with sed; uploads use the single-flag pattern (--upload "${PAYLOAD_DIR}:/sandbox") with files referenced at the basename-prefixed path that #952 / #1028 established. - README documents the trust model honestly: structured rule is the contract, agent rationale is a hint, prover validation badge in progress per RFC 0001. Move the deterministic no-LLM regression harness out of examples/ into e2e/policy-advisor/ -- it was a parallel demo, not an example. Same loop without the LLM, useful for iterating on the proxy and policy.local API. * style(sandbox,cli): apply rustfmt Whitespace-only fixups caught by mise run pre-commit. No functional change. * perf(examples): cap Codex reasoning at 'low' in policy demo The demo task is mechanical (one HTTP request, parse a structured 403, post a JSON proposal, retry). Codex's default high-effort reasoning roughly doubles the demo's wall time without improving outcomes; running at 'low' lands the same minimal L7 grant in roughly half the time. Override with DEMO_CODEX_REASONING=medium (or higher) to compare runs. * fix(sandbox): harden policy.local denials endpoint Three changes addressing review feedback before merging the agent-driven policy management MVP: - Distinguish "OCSF JSONL enabled, no denials" from "OCSF JSONL disabled, nothing to read." The endpoint now returns a `log_available` flag and an explanatory `note` when the log file is missing, so the in-sandbox agent can give the developer an accurate hint instead of a misleading empty list. - Stop echoing the OCSF `message` field in the per-denial summary. The proxy's denial messages can include the request path with query string (e.g., `?access_token=...`); the structured `host`/`port`/`method`/ `path`/`binary` fields carry everything the agent needs to draft a proposal, and `path` is sourced from `http_request.url.path` which already excludes the query string. - Cap `read_request_body` at a 15s timeout. Bounds slowloris-style stalls from a misbehaving in-sandbox process. The proxy listener only accepts loopback connections so practical impact is small, but this is cheap defense-in-depth. New tests cover the missing-log signal and the message-redaction guarantee. * fix(examples): redact tokens in agent log tail and validate DEMO_FILE_DIR Two small hardening passes on the policy management demo: - `fail()` now pipes the agent log tail through a redactor that masks the GitHub token and Codex credential triple before printing. Codex itself is well-behaved about not echoing the token, but a misbehaving tool call could leak it; this is a final safety net before the log hits the developer's terminal (and any clipboard or chat history that follows). - `validate_env` now regex-checks DEMO_FILE_DIR with the same allow-list the other path-shaped variables use. The value is interpolated through sed with `|` as the delimiter when rendering the agent task; rejecting unsupported characters keeps the templating predictable and stops a user-supplied value from breaking out into a shell context. * refactor(sandbox): centralize policy.local routes and skill path Addresses review feedback that the deny body's `next_steps` array and the route table could drift apart. The route paths and skill location now live as `pub const`s in `policy_local.rs` and feed both: - the dispatcher in `route_request` that matches against them - a new `agent_next_steps()` helper that builds the JSON the L7 deny body embeds `l7/rest.rs::deny_response_body` calls `policy_local::agent_next_steps()` instead of inlining the array, so adding or renaming a route is a one-line change in `policy_local.rs` and the agent contract follows automatically. * feat(sandbox): switch /v1/denials to shorthand log pass-through Previously /v1/denials parsed `/var/log/openshell-ocsf.*.log` (OCSF JSONL) and returned structured per-event objects. JSONL is opt-in via `ocsf_json_enabled`, so the endpoint returned an empty list with a "log not enabled" hint by default — agents had to navigate a setup step before the inspect-recent-denials guidance was useful. Switch to reading the shorthand log at `/var/log/openshell.*.log`, which is always-on and the same human-readable format `openshell logs` displays. The endpoint now returns raw shorthand lines (newest first) — the agent reads them directly, no field parsing. Tradeoffs: - Removes the JSONL-on-by-default debate: shorthand is already on, no defaults change. - Updating shorthand is a single-file change in this repo; no schema rev needed when we want to add fields. Implementation: - `read_recent_denial_lines` walks shorthand log files newest-first, filters lines with ` OCSF ` AND ` DENIED ` (the OCSF action label, uppercase, space-bounded). - `collect_shorthand_log_files` matches `openshell.<date>.log`; the trailing dot in `SHORTHAND_LOG_PREFIX = "openshell."` excludes `openshell-ocsf.<date>.log` so JSONL-on doesn't bleed into responses. - 4096-byte cap per surfaced line as defense against pathological inputs. - Skill doc updated to reflect that `/v1/denials` returns raw shorthand lines, not structured fields. Defense-in-depth on query-string secrets: - `redact_query_strings` strips `?<query>` to `?[redacted]` from each surfaced line. The L7 relay path emits OCSF events using `redacted_target` (secret-placeholder redaction), but the FORWARD deny path in `proxy.rs` populates `OcsfUrl::new("http", host, path, port)` and `.message(...)` with the raw request path — query string included. Stripping queries at the consumer guards `/v1/denials` regardless of whether the upstream emit sites are tightened. The on-disk log is not rewritten by this change; that is a separate hardening task tracked for the FORWARD path emit sites in proxy.rs. - `truncate_at_char_boundary` is UTF-8 safe; redaction runs before truncation so a cut cannot slice mid-secret. Tests: - `recent_denials_returns_newest_first_from_shorthand_lines` covers the happy path with mixed allowed/denied/non-OCSF lines. - `recent_denials_skips_jsonl_log_files` confirms JSONL files don't surface even if present. - `recent_denials_truncates_pathological_lines` covers the cap. - `is_ocsf_denial_line_filters_correctly` covers the line-level filter. - `redact_query_strings_removes_query_from_url_token` and `redact_query_strings_removes_query_in_reason_tag` cover the redaction in both URL token and `[reason:...]` contexts. - `truncate_at_char_boundary_does_not_panic_on_multibyte` covers the UTF-8 safety. * chore(sandbox): align proto inits with main's L7 GraphQL additions Post-rebase fixups after #1083 (GraphQL L7 inspection) landed on main and introduced new fields on the proto types this branch constructs: - `crates/openshell-sandbox/src/l7/relay.rs`: two `deny_with_redacted_target` call sites (REST and GraphQL relay deny paths) now pass the `DenyResponseContext` argument that `rest::send_deny_response` expects. Both sites pass `host`, `port`, and `binary` from the existing `L7EvalContext`, matching the pattern used at the primary deny site. - `crates/openshell-sandbox/src/policy_local.rs`: `L7Allow`, `L7DenyRule`, and `NetworkEndpoint` proto initializers now populate the new GraphQL and path-scoping fields with empty defaults. Agent-authored proposals via `policy.local` target REST/SQL/L4 today; GraphQL operation matching is set on the gateway side or via direct YAML, so empty defaults are correct here. No behavior change. `cargo test -p openshell-sandbox --lib` (650 tests) and `cargo clippy -p openshell-sandbox --lib --tests -- -D warnings` clean. * feat(sandbox): gate agent policy proposals behind opt-in feature flag The agent-driven policy proposal surface delivered by this PR (skill install, `policy.local` API, `next_steps` array on L7 deny bodies) is now opt-in via the new `agent_policy_proposals_enabled` setting. Default false. Same shape as `providers_v2_enabled`: registered in `openshell-core::settings`, sandbox-level, hot-toggleable via the existing settings poll loop. Why: the surface is a novel agent-controlled mutation point in every sandbox. The per-proposal developer approval gate is a correctness control, but it doesn't address "should this sandbox have an agent-authoring API at all" — compliance teams may want that question closed. The flag is the second gate. Implementation: - New registry entry + `AGENT_POLICY_PROPOSALS_ENABLED_KEY` constant in `openshell-core::settings`. - `lib.rs`: process-wide `OnceLock<Arc<AtomicBool>>` mirroring the `OCSF_CTX` pattern. `agent_proposals_enabled()` is the single read point. - Initial settings fetch added to `run_sandbox` so skill install honors the flag at startup (not just on the poll loop's first tick). - Skill install in `run_sandbox` is gated on the flag. - `policy_local::route_request` returns `404 feature_disabled` for all routes when the flag is off — including the otherwise-public `current_policy` and `denials` routes. When the surface is off it's off entirely. - `policy_local::agent_next_steps` returns an empty array when the flag is off so deny bodies don't advertise routes that 404. - Poll loop updates the atomic on each tick, lazily installs the skill on a false→true transition (no claw-back on true→false; stale skill on disk is harmless because route + next_steps gate on the live atom). Tests: - Shared `test_helpers::ProposalsFlagGuard` mutex+atomic guard for the process-wide flag, used across `policy_local::tests` and `l7::rest::tests`. - New: `agent_next_steps_returns_empty_when_flag_off`, `agent_next_steps_returns_full_array_when_flag_on`, `route_request_returns_feature_disabled_when_flag_off`. - Updated existing tests that exercise the deny body or the route dispatcher to set the flag on first. - Full sandbox lib test suite: 653 pass, clippy clean. Demo and e2e: - `examples/agent-driven-policy-management/demo.sh` and `e2e/policy-advisor/test.sh` now snapshot the prior global value of the setting, set it to true before sandbox creation (so the supervisor's initial poll picks it up), and restore on exit (delete if previously unset, otherwise write the prior value back). Docs: - RFC 0001 MVP-implementation note documents the flag, default, and intended soft-launch posture. * test(policy-advisor): require proposal opt-in for e2e * refactor(sandbox): group policy poll loop state * test(e2e): isolate Kubernetes user namespace test --------- Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Taylor Mutch
|
316c788eac |
fix(helm): derive grpcEndpoint from chart context (#1241)
* fix(helm): derive grpcEndpoint from chart context The chart hardcoded server.grpcEndpoint to https://openshell.openshell.svc.cluster.local:8080, which only matched the in-cluster Service DNS for the standard release name and namespace. A new helper now builds <scheme>://<fullname>.<namespace>.svc.cluster.local:<port> from chart context, picking the scheme from server.disableTls. An explicit server.grpcEndpoint override is passed through verbatim. * chore(scripts): validate k3d cluster name length early helm-k3s-local.sh derives the cluster name from the current branch suffix. Long branch names produced names exceeding k3d's 32-char cap and failed deep inside k3d cluster create with a confusing validation error. cmd_create now bails out before invoking docker/k3d with a copy-pasteable HELM_K3S_CLUSTER_NAME override hint. Status, start, stop, delete, and help remain unaffected so an over-long derived name does not block diagnostics. |
||
|
Drew Newberry
|
70a0f6c547 | refactor(cli): remove gateway lifecycle management (#1221) | ||
|
Alexander Watson
|
142a3a3d31 |
fix(examples): harden multi-agent notepad 409 retry and improve docs (#1166)
* fix(examples): harden multi-agent notepad 409 retry and improve docs Add decorrelated jitter to the GitHub Contents API retry loop so racing workers don't retry in lockstep, and widen the retry set to 5xx for transient server errors. Without jitter, N concurrent workers trip the same backoff schedule and re-collide on each beat. Also: - Extract the inline 220-line sandbox runner from demo.sh into a separate runner.sh. demo.sh stays as host orchestration; runner.sh is the per-sandbox program. demo.sh shrinks 460 to 240 lines. - Rewrite README to lead with "Why GitHub as a shared notepad?" and add a "Memory architecture variants" section covering pile, append-journal, and indexed-memory patterns. - Switch Quick Start to use \`gh auth token\` as the primary path. - Pass the worker slice arg explicitly instead of reading \$9 from the outer dispatcher. - Add .gitignore for personal run_demo.sh helpers. * docs updates |
||
|
Drew Newberry
|
f56c09c7df | docs: update gateway deployment architecture (#1108) | ||
|
Alexander Watson
|
25c4fdecd1 | fix(examples): repair multi-agent notepad uploads (#1152) | ||
|
Drew Newberry
|
2e0afeabe1 | feat(vm): derive guest rootfs from sandbox images (#957) | ||
|
Alexander Watson
|
ebbd9dee5a |
docs(examples): add multi-agent notepad demo (#991)
Signed-off-by: Alexander Watson <zredlined@gmail.com> |
||
|
Derek Carr
|
de9dce0436 |
fix(e2e): use high UID range to avoid host user conflicts (#978)
Change sandbox user UID from 1000 to 1000660000 in custom image examples and E2E tests. Using a high UID range (1000000000+) prevents conflicts with host users when running without user namespace remapping, where container UIDs map directly to host UIDs. This resolves fork failures caused by RLIMIT_NPROC enforcement when the host user already has many threads running. Signed-off-by: Derek Carr <decarr@redhat.com> |
||
|
Piotr Mlocek
|
df38d1f66f | feat(ci): add Markdown and Mermaid linting (#933) | ||
|
Vinicius Corrêa
|
8c4b17221c |
Missed input parameter (#645)
Missed "--from openclaw" input parameter. Without this input sandbox container didn't install "openclaw": 'Stop with: openshell forward stop 18789 eager-dragonfly '/bin/bash: line 1: openclaw-start: command not found |
||
|
John T. Myers
|
f37b69b5e5 |
feat(sandbox): auto-detect TLS and terminate unconditionally for credential injection (#544)
* feat(sandbox): auto-detect TLS and terminate unconditionally for credential injection Closes #533 The proxy now auto-detects TLS by peeking the first bytes of each connection. When TLS is detected, it terminates unconditionally — enabling credential injection and optional L7 inspection without requiring explicit 'tls: terminate' in the policy. |
||
|
Alexander Watson
|
a0aea6904b | Added pauses and syntax highlighting to demo for clarity (#358) | ||
|
John T. Myers
|
d3e1b31db9 |
feat(sandbox): log connection attempts that bypass proxy path (#326)
* feat(sandbox): log connection attempts that bypass proxy path Add iptables LOG + REJECT rules inside the sandbox network namespace to detect and diagnose direct connection attempts that bypass the HTTP CONNECT proxy. This provides two improvements: 1. Fast-fail UX: applications get immediate ECONNREFUSED instead of a 30-second timeout when they bypass the proxy 2. Diagnostics: a /dev/kmsg monitor emits structured BYPASS_DETECT tracing events with destination, protocol, process identity, and actionable hints Both TCP and UDP bypass attempts are covered (UDP catches DNS bypass). The feature degrades gracefully if iptables or /dev/kmsg are unavailable. Closes #268 * chore: track .python-version to pin Python 3.13.12 for uv The sandbox base image runs Python 3.13. A stale venv on 3.12 causes all exec_python E2E tests to fail because cloudpickle bytecode is not compatible across minor versions. * fix(cluster): preserve hostGatewayIP across fast deploys The fast deploy's helm upgrade was missing the hostGatewayIP value that the bootstrap entrypoint injects into the HelmChart CR. This caused host.openshell.internal hostAliases to be lost from the gateway pod and sandbox pods after any fast deploy, breaking host gateway routing. Read the IP from the HelmChart CR and pass it through to helm upgrade. * wip: fix iptables path resolution, use dmesg for kmsg, add CAP_SYSLOG * fix(sandbox): restore NetworkNamespace Drop impl, remove dead kmsg code The Drop impl for NetworkNamespace was accidentally deleted during the bypass detection refactor, which would cause network namespaces and veth interfaces to leak on every sandbox shutdown. Also removes dead kmsg volume/mount code (bypass monitor uses dmesg instead of direct /dev/kmsg access) and removes an accidentally committed session transcript file. |
||
|
Drew Newberry
|
0a7ffe1669 | ci(release): gate python wheels on e2e for tagged releases (#319) | ||
|
Alexander Watson
|
158c925342 |
docs(examples): add sandbox policy quickstart walkthrough (#266)
* docs(examples): add sandbox policy quickstart walkthrough Add an interactive getting-started example that demonstrates OpenShell's network policy system end-to-end: default-deny, L7 read-only access, and audit logging — all with a single YAML policy file. - examples/sandbox-policy-quickstart/policy.yaml: policy with default static fields (filesystem, landlock, process) so it works out of the box with `openshell policy set` - examples/sandbox-policy-quickstart/demo.sh: automated demo script using printf (portable) and openshell ssh-proxy for sandbox exec - examples/sandbox-policy-quickstart/README.md: step-by-step manual walkthrough - README.md: add "See network policy in action" section linking to the quickstart Signed-off-by: Alexander Watson <zredlined@gmail.com> Made-with: Cursor Signed-off-by: Alexander Watson <zredlined@gmail.com> Made-with: Cursor * docs: soften default-deny wording to minimal outbound access Sandbox defaults vary by type and community configs, so "all outbound traffic is blocked" is too absolute. Made-with: Cursor * docs: use curl -sS so L4 deny errors are visible curl -s suppresses stderr, hiding the 403 from the CONNECT proxy. Adding -S ensures the error message is always shown. Made-with: Cursor --------- Signed-off-by: Alexander Watson <zredlined@gmail.com> |
||
|
Drew Newberry
|
fbd93a4632 | refactor: rename navigator- crate prefix to openshell- (#277) | ||
|
Drew Newberry
|
7b0a243304 | ci: remove sandbox docker build from publish and e2e workflows (#275) | ||
|
Drew Newberry
|
89d21d7852 | refactor(sandbox): sandboxes are managed as separate community images (#267) | ||
|
Piotr Mlocek
|
14e296d318 | fix(cli): add --no-keep for ephemeral sandbox create cleanup (#258) | ||
|
John T. Myers
|
2e4c2fcc84 |
fix(proxy): stream inference responses instead of buffering entire body (#261)
* fix(proxy): stream inference responses instead of buffering entire body The inference.local proxy path called response.bytes().await which buffered the entire upstream response before sending anything to the client. For streaming SSE responses this inflated TTFB from sub-second to the full generation time, causing clients with TTFB timeouts to abort. Add a streaming proxy variant that returns response headers immediately and forwards body chunks incrementally using HTTP chunked transfer encoding. Non-streaming responses and mock routes continue to work through the existing buffered path. Closes #260 * docs: update architecture docs and example for inference streaming * test(example): add direct NVIDIA endpoint tests via L7 TLS intercept Expand inference example to 4 test cases: inference.local and direct endpoint, each streaming and non-streaming. The direct path exercises the L7 REST relay (relay_chunked) to verify it already streams correctly. NVIDIA_API_KEY is picked up from the sandbox env when started with --provider nvidia. |
||
|
John T. Myers
|
454327d890 |
feat(policy): add policy recommendation plumbing (#204) (#222)
* feat(policy): add policy recommendation plumbing — denial aggregation, transport, approval pipeline, and mechanistic recommendations Implement the infrastructure layer for automated policy recommendations (#204): - Proto: 9 new RPCs and messages for draft policy lifecycle (submit, get, approve, reject, approve-all, edit, undo, clear, history) - Persistence: SQLite/Postgres migrations and store methods for draft_policy_chunks and denial_summaries tables - Server: Full gRPC handler implementations with mechanistic mapper that auto-generates NetworkPolicyRule proposals from denial summaries - Sandbox: DenialAggregator with MPSC channel, deduplication, periodic flush to gateway via SubmitPolicyAnalysis - CLI: 'openshell draft' subcommand with get/approve/reject/approve-all/undo/clear/history operations - TUI: Draft recommendations panel accessible from sandbox policy view - Docs: Architecture documentation in architecture/policy-advisor.md * feat(policy): add L7-aware mechanistic mapper and policy advisor CTF example Add L7 rule generation to mechanistic mapper (build_l7_rules, generalise_path, looks_like_id) with 3 new unit tests. Add examples/policy-advisor/ with a 7-gate CTF script, restrictive sandbox policy, and walkthrough README. * fix(policy): use sandbox name for denial flush and add TUI draft badges Fix denial aggregator passing sandbox UUID instead of name to SubmitPolicyAnalysis, which caused 'sandbox not found' errors on flush. Add notification badges to the TUI sandbox list and detail header showing pending draft recommendation counts. * fix(policy): deduplicate draft chunks and tolerate overlapping OPA rules Skip draft chunk creation when a pending/approved chunk already covers the same host:port endpoint, preventing duplicate rules across denial aggregator flush cycles. Rewrite three OPA complete rules (network_policy_for_request, matched_network_policy, matched_endpoint_config) to tolerate multiple matching policies without triggering a "complete rule conflict" error. network_policy_for_request becomes a boolean, matched_network_policy uses a set comprehension with min(), and matched_endpoint_config uses an array comprehension with index-0 selection. * feat(tui): interactive draft actions, highlight bar, and detail popup Rework the draft recommendations panel to match the logs UX: - Highlight bar (green accent + background) instead of arrow marker - Viewport-aware j/k scrolling with g/G for top/bottom - Enter opens a full-screen detail popup showing endpoints, binaries, rationale, security notes, and action hints Add approve/reject/approve-all draft actions: - [a] approve selected chunk, [x] reject, [A] approve all pending - Actions work from both the list view and the detail popup - gRPC calls run async; result updates status bar and refreshes data - Nav bar shows all available keybindings Fix draft count refresh: sandbox_draft_counts now refreshes on every tick (not just Dashboard), so the detail header badge updates in real time. Improve badge labels: show 'N pending' instead of a bare number in both the dashboard sandbox list and sandbox detail header. * refactor(policy): DB-level draft chunk dedup with hit counter and timestamps Replace the in-memory HashSet dedup in SubmitPolicyAnalysis with a database-level upsert. New denormalized columns on draft_policy_chunks: - host, port: extracted from proposed_rule at insert time - hit_count: incremented on conflict (same sandbox + host + port) - first_seen_ms, last_seen_ms: track when the endpoint was first and most recently proposed A partial unique index (WHERE status IN ('pending','approved')) ensures only one active chunk per endpoint per sandbox; rejected/superseded chunks don't block new proposals. Surface hit_count and first/last_seen in: - CLI: 'openshell draft get' shows 'Hits: N (first ..., last ...)' - TUI: detail popup shows hits row; list view shows 'Nx' suffix * fix(policy): optimistic retry on policy version conflicts + structured logging merge_chunk_into_policy and remove_chunk_from_policy now retry up to 5 times on UNIQUE constraint violations (version conflicts from concurrent approvals). Each attempt re-reads the latest policy, re-merges the rule, and increments the version. This eliminates the race condition where rapid successive approvals would fail with a DB error. Add structured tracing to all draft action handlers: - ApproveDraftChunk: logs rule_name, host, port, hit_count before merge and version + policy_hash after success - RejectDraftChunk: logs rule_name, host, port, reason - ApproveAllDraftChunks: logs pending_count at start, per-chunk merge progress, and final summary with chunks_approved/skipped - UndoDraftChunk: logs before/after with rule_name and version - Retry attempts log as warnings with attempt number and conflicting version * wip: forward proxy fix, mapper allowed_ips, TUI polish, CTF rewrite * fix(tui): use correct --gateway flag for ssh-proxy ProxyCommand * chore: add Docker cleanup script for stale images, volumes, and build cache * feat(tui): approve-all confirmation modal and CTF cleanup Add [A] confirmation popup that snapshots pending chunks, shows a scrollable list, and approves each chunk individually on confirm. This prevents approving chunks that arrived after the modal opened. Remove transient issue #205 reference from CTF victory banner. * fix(tui): correct import ordering for rustfmt * wip: stateful toggle model, rename to network rules Draft chunks now follow a toggle state machine: pending -> approved | rejected (initial decision) approved <-> rejected (toggle) One row per (sandbox_id, host, port) via expanded unique index. Rejecting an approved rule removes it from the active policy. Re-approving a rejected rule merges it back. Rename CLI from 'draft' to 'rule', TUI from 'Draft Recommendations' to 'Network Rules'. State-aware keybindings: approved shows [x] Revoke, rejected shows [a] Approve. Fix sandbox detail hiding delete confirmation behind pending message. * refactor(policy): move mapper sandbox-side, slim schema, per-binary granularity Move mechanistic mapper from gateway to sandbox so all analysis runs sandbox-side (N sandboxes = N independent pipelines). Gateway is now a thin validate + persist + approval layer. Architectural changes: - Move mechanistic_mapper.rs from navigator-server to navigator-sandbox - Sandbox flush flow: aggregator drains -> mapper runs -> proposals sent - Gateway SubmitPolicyAnalysis: validate + persist only, no mapper - Drop denial_summaries table (write-only, zero readers) - Consolidate migrations 003+004+005 into single 003 Schema slimming: - Drop 5 unused columns from draft_policy_chunks (stage, denial_refs, supersedes_chunk_id, analysis_mode, decided_by) - Add per-binary granularity: binary column, widen unique index to (sandbox_id, host, port, binary) - Mapper groups by (host, port, binary), one proposal per triple - Merge appends binary to existing rule; revoke removes just that binary CTF & UX: - 7-gate CTF: add Gate 3 (curl -> ifconfig.me:80) for per-binary demo - TUI shows binary short name in list, full path in detail popup - CLI output shows binary field - Idempotent rule names, hit_count accumulates real denial counts - Rationale text no longer bakes in stale denial count |
||
|
Drew Newberry
|
95d7ae077e | refactor(cli): remove kubeconfig port, add doctor llm-help, update debug docs (#252) | ||
|
Drew Newberry
|
bcc6dad1d0 | feat(cli): launch sandbox editors via managed ssh include (#226) | ||
|
Drew Newberry
|
d6c6e97679 | chore: remove navigator references from codebase (#208) | ||
|
Drew Newberry
|
984d1a6e5c | chore: rename project from NemoClaw to OpenShell (#198) | ||
|
Piotr MlocekandMiyoung Choi
|
107c85d1d7 |
docs(inference): clarify local inference routing (#190)
* docs(inference): clarify local inference routing * docs(inference): update provider and model examples * fix doc build --------- Co-authored-by: Miyoung Choi <miyoungc@nvidia.com> |
||
|
Drew Newberry
|
a2de1f24e5 | feat: add Cloudflare tunnel auth support (#178) | ||
|
Drew Newberry
|
ed53c35d2b | feat(cli): improve sandbox provisioning status messages and UX (#175) | ||
|
Drew Newberry
|
07b9d5d00e | feat(cli): restructure CLI commands for simpler UX (#156) | ||
|
Piotr Mlocek
|
31d7ca53ff |
refactor(inference): simplify routing — introduce inference.local, remove implicit catch-all (#146)
Remove multi-route CRUD system and replace with single managed cluster route (inference.local). Key changes: - Remove inference route CRUD RPCs and CLI commands - Remove InspectForInference OPA action; policy is binary allow/deny - Introduce AuthHeader enum and InferenceProviderProfile in navigator-core - Router is now provider-agnostic: auth style carried on ResolvedRoute - Replace InferenceRouteSpec with ClusterInferenceConfig (2 fields vs 8) - Rename proto: routing_hint->name, SandboxResolvedRoute->ResolvedRoute, GetSandboxInferenceBundle->GetInferenceBundle, drop sandbox_id param - Rename RouteConfig.route -> RouteConfig.name; use inference.local - Add 'nemoclaw cluster inference update' for partial config changes - Delete stale navigator.inference.v1.rs checked-in proto file - Update architecture docs, agent skills, and CLI reference Closes #133 |
||
|
Drew Newberry
|
3da64744f0 | feat(cli): add --from flag to sandbox create for unified image sources (#89) |