* feat(sandbox): switch device plugin to CDI injection mode
Configure the NVIDIA device plugin to use deviceListStrategy=cdi-cri so
that GPU devices are injected via direct CDI device requests in the CRI.
Sandbox pods now only require the nvidia.com/gpu resource request —
runtimeClassName is no longer set on GPU pods.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* feat(bootstrap): switch GPU injection to CDI where supported
Use an explicit CDI device request (driver="cdi", device_ids=["nvidia.com/gpu=all"])
when the Docker daemon reports CDI spec directories via GET /info (SystemInfo.CDISpecDirs).
This makes device injection declarative and decouples spec generation from consumption.
When the daemon reports no CDI spec directories, fall back to the legacy NVIDIA device
request (driver="nvidia", count=-1) which relies on the NVIDIA Container Runtime hook.
Failure modes for both paths are equivalent: a missing or stale NVIDIA Container Toolkit
installation will cause container start to fail.
CDI spec generation is out of scope for this change; specs are expected to be
pre-generated out-of-band, for example by the NVIDIA Container Toolkit.
---------
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Co-authored-by: Piotr Mlocek <pmlocek@nvidia.com>
* feat(sandbox): add L7 query parameter matchers
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* fix(sandbox): decode + as space in query params and validate glob syntax
Three improvements from PR #617 review:
1. Decode + as space in query string values per the
application/x-www-form-urlencoded convention. This matches Python's
urllib.parse, JavaScript's URLSearchParams, Go's url.ParseQuery, and
most HTTP frameworks. Literal + should be sent as %2B.
2. Add glob pattern syntax validation (warnings) for query matchers.
Checks for unclosed brackets and braces in glob/any patterns. These
are warnings (not errors) because OPA's glob.match is forgiving,
but they surface likely typos during policy loading.
3. Add missing test cases: empty query values, keys without values,
unicode after percent-decoding, empty query strings, and literal +
via %2B encoding.
* fix(sandbox): add missing query_params field in forward proxy L7 request info
* style(sandbox): fix formatting in proxy L7 query param parsing
---------
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
* fix(sandbox): handle per-path Landlock errors instead of abandoning entire ruleset
A single missing path (e.g., /app in containers without that directory)
caused PathFd::new() to propagate an error out of the entire Landlock
setup closure. Under BestEffort mode, this silently disabled all
filesystem restrictions for the sandbox.
Changes:
- Extract try_open_path() and classify_path_error() helpers that handle
PathFd failures per-path instead of per-ruleset
- BestEffort mode: skip inaccessible paths with a warning, apply
remaining rules
- HardRequirement mode: fail immediately on any inaccessible path
- Add zero-rule safety check to prevent applying an empty ruleset that
would block all filesystem access
- Pre-filter system-injected baseline paths (e.g., /app) in enrichment
functions so missing paths never reach Landlock
- Add unit tests for try_open_path, classify_path_error, and error
classification for ENOENT, EACCES, ELOOP, ENAMETOOLONG, ENOTDIR
- Update user-facing docs and architecture docs with Landlock behavior
tables, baseline path filtering, and compatibility mode semantics
- Fix stale ABI::V1 references in docs (code uses ABI::V2)
Closes#664
* fix(sandbox): use debug log for NotFound in Landlock best-effort mode
NotFound errors for stale baseline paths (e.g. /app persisted in the
server-stored policy but absent in this container) are expected in
best-effort mode. Downgrade from warn! to debug! so the message does
not leak into SSH exec stdout (the pre_exec hook inherits the tracing
subscriber whose writer targets fd 1).
Genuine errors (permission denied, symlink loops, etc.) remain at warn!
for operator visibility.
Also move custom_image e2e marker from /opt to /etc (a Landlock baseline
read-only path) since the security fix now properly enforces filesystem
restrictions.
---------
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
* refactor(l7): export evaluate_l7_request for cross-module use
Make evaluate_l7_request() public so the forward proxy path can
evaluate individual requests against L7 policy without going
through the full relay_with_inspection() loop.
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
* fix(proxy): add L7 inspection to forward proxy path
The forward proxy previously rejected all requests to endpoints with
L7 rules (blanket 403), forcing clients through the CONNECT tunnel.
This meant policies like read-only (allow GET, block POST) had no
effect on plain http:// requests through the forward proxy.
Replace the blanket rejection with actual L7 evaluation:
- Query L7 config for the endpoint (same as before)
- Clone the OPA engine and evaluate the request method/path
- Allow if L7 policy permits, deny with 403 if enforcement is enforce
- Audit mode: log but allow (matching CONNECT path behavior)
- Fail-closed: deny on evaluation errors
The forward proxy uses Connection: close (one request per connection),
so a single evaluation suffices — no relay loop needed.
Update e2e tests to validate the new behavior:
- GET /allowed → 200 (L7 policy allows)
- POST /allowed → 403 (L7 policy denies, enforcement: enforce)
Update security-policy.md to reflect the new forward proxy L7 behavior.
Closes#643
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
* style(proxy): apply cargo fmt formatting
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
---------
Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
Co-authored-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
* feat(sandbox): auto-detect TLS and terminate unconditionally for credential injection
Closes#533
The proxy now auto-detects TLS by peeking the first bytes of each
connection. When TLS is detected, it terminates unconditionally —
enabling credential injection and optional L7 inspection without
requiring explicit 'tls: terminate' in the policy.
Disables GPU Feature Discovery and Node Feature Discovery DaemonSets and
overrides the device plugin's default nodeAffinity to null so it schedules
unconditionally on the single-node gateway without requiring NFD/GFD labels.
Setting affinity to an empty map ({}) does not override the chart defaults
because Helm deep-merges user values with chart defaults. Using null explicitly
removes the key, causing the chart template to skip the affinity block entirely.
* feat(policy): support host wildcards and multi-port endpoints
Add glob-style host wildcards to endpoints[].host using OPA's
glob.match with "." as delimiter — *.example.com matches a single
DNS label, **.example.com matches across labels. Validation rejects
bare * and requires *. prefix; warns on broad patterns like *.com.
Add repeated uint32 ports field to NetworkEndpoint for multi-port
support. Backwards compatible: existing port scalar is normalized to
ports array. Both the proto-to-JSON and YAML-to-JSON conversion paths
emit a ports array; Rego always references endpoint.ports[_].
Fix OpaEngine::reload() to route through the full preprocessing
pipeline instead of bypassing L7 validation and port normalization.
Closes#359
* fix(policy): reject configs with both port and ports set
---------
Co-authored-by: johntmyers <johntmyers@users.noreply.github.com>
* feat(sandbox): log connection attempts that bypass proxy path
Add iptables LOG + REJECT rules inside the sandbox network namespace to
detect and diagnose direct connection attempts that bypass the HTTP
CONNECT proxy. This provides two improvements:
1. Fast-fail UX: applications get immediate ECONNREFUSED instead of a
30-second timeout when they bypass the proxy
2. Diagnostics: a /dev/kmsg monitor emits structured BYPASS_DETECT
tracing events with destination, protocol, process identity, and
actionable hints
Both TCP and UDP bypass attempts are covered (UDP catches DNS bypass).
The feature degrades gracefully if iptables or /dev/kmsg are unavailable.
Closes#268
* chore: track .python-version to pin Python 3.13.12 for uv
The sandbox base image runs Python 3.13. A stale venv on 3.12 causes
all exec_python E2E tests to fail because cloudpickle bytecode is not
compatible across minor versions.
* fix(cluster): preserve hostGatewayIP across fast deploys
The fast deploy's helm upgrade was missing the hostGatewayIP value that
the bootstrap entrypoint injects into the HelmChart CR. This caused
host.openshell.internal hostAliases to be lost from the gateway pod
and sandbox pods after any fast deploy, breaking host gateway routing.
Read the IP from the HelmChart CR and pass it through to helm upgrade.
* wip: fix iptables path resolution, use dmesg for kmsg, add CAP_SYSLOG
* fix(sandbox): restore NetworkNamespace Drop impl, remove dead kmsg code
The Drop impl for NetworkNamespace was accidentally deleted during the
bypass detection refactor, which would cause network namespaces and
veth interfaces to leak on every sandbox shutdown.
Also removes dead kmsg volume/mount code (bypass monitor uses dmesg
instead of direct /dev/kmsg access) and removes an accidentally
committed session transcript file.
Closes#273
Verify inference endpoints synchronously on the server during set/update, expose a --no-verify escape hatch in the CLI and Python helper, and return actionable failures when validation does not pass.
* feat(sandbox): inject host gateway hostAliases into sandbox pods
Sandbox pods running in the k3s cluster cannot resolve host.docker.internal
by default, preventing them from reaching services on the Docker host.
Detect the host gateway IP (default route) in the cluster entrypoint,
thread it through the Helm chart to the gateway server, and inject
hostAliases entries (host.docker.internal, host.openshell.internal)
into every sandbox pod spec. The injection is conditional -- when the
IP is empty (non-Docker deployments), no hostAliases are added.
---------
Co-authored-by: Piotr Mlocek <pmlocek@nvidia.com>
PR #281 removed the shared openshell-cluster Docker network in favor of
the default bridge. This restores custom bridge networking but makes each
gateway use its own isolated network named openshell-cluster-{name},
matching the existing container/volume naming convention.
Changes:
- Add network_name() to constants.rs for per-gateway network naming
- Add ensure_network() with retry/backoff and force_remove_network()
parameterized by network name instead of a global constant
- Attach containers to their per-gateway network via network_mode
- Disconnect and remove the network during gateway destroy
- Wire ensure_network() into the deploy flow before ensure_volume()
- Update architecture docs to reflect per-gateway network isolation
* feat(sandbox): add gpu sandbox scheduling support
Allow sandbox creation to request GPU resources explicitly or infer them from GPU image names. This wires GPU intent through bootstrap, validates gateway support, and adds dedicated GPU E2E coverage for follow-up cluster testing.
* fix(proxy): stream inference responses instead of buffering entire body
The inference.local proxy path called response.bytes().await which
buffered the entire upstream response before sending anything to the
client. For streaming SSE responses this inflated TTFB from sub-second
to the full generation time, causing clients with TTFB timeouts to abort.
Add a streaming proxy variant that returns response headers immediately
and forwards body chunks incrementally using HTTP chunked transfer
encoding. Non-streaming responses and mock routes continue to work
through the existing buffered path.
Closes#260
* docs: update architecture docs and example for inference streaming
* test(example): add direct NVIDIA endpoint tests via L7 TLS intercept
Expand inference example to 4 test cases: inference.local and direct
endpoint, each streaming and non-streaming. The direct path exercises
the L7 REST relay (relay_chunked) to verify it already streams
correctly. NVIDIA_API_KEY is picked up from the sandbox env when
started with --provider nvidia.
* feat(policy): add policy recommendation plumbing — denial aggregation, transport, approval pipeline, and mechanistic recommendations
Implement the infrastructure layer for automated policy recommendations (#204):
- Proto: 9 new RPCs and messages for draft policy lifecycle (submit, get, approve, reject, approve-all, edit, undo, clear, history)
- Persistence: SQLite/Postgres migrations and store methods for draft_policy_chunks and denial_summaries tables
- Server: Full gRPC handler implementations with mechanistic mapper that auto-generates NetworkPolicyRule proposals from denial summaries
- Sandbox: DenialAggregator with MPSC channel, deduplication, periodic flush to gateway via SubmitPolicyAnalysis
- CLI: 'openshell draft' subcommand with get/approve/reject/approve-all/undo/clear/history operations
- TUI: Draft recommendations panel accessible from sandbox policy view
- Docs: Architecture documentation in architecture/policy-advisor.md
* feat(policy): add L7-aware mechanistic mapper and policy advisor CTF example
Add L7 rule generation to mechanistic mapper (build_l7_rules,
generalise_path, looks_like_id) with 3 new unit tests. Add
examples/policy-advisor/ with a 7-gate CTF script, restrictive
sandbox policy, and walkthrough README.
* fix(policy): use sandbox name for denial flush and add TUI draft badges
Fix denial aggregator passing sandbox UUID instead of name to
SubmitPolicyAnalysis, which caused 'sandbox not found' errors on
flush. Add notification badges to the TUI sandbox list and detail
header showing pending draft recommendation counts.
* fix(policy): deduplicate draft chunks and tolerate overlapping OPA rules
Skip draft chunk creation when a pending/approved chunk already covers
the same host:port endpoint, preventing duplicate rules across denial
aggregator flush cycles.
Rewrite three OPA complete rules (network_policy_for_request,
matched_network_policy, matched_endpoint_config) to tolerate multiple
matching policies without triggering a "complete rule conflict" error.
network_policy_for_request becomes a boolean, matched_network_policy
uses a set comprehension with min(), and matched_endpoint_config uses
an array comprehension with index-0 selection.
* feat(tui): interactive draft actions, highlight bar, and detail popup
Rework the draft recommendations panel to match the logs UX:
- Highlight bar (green accent + background) instead of arrow marker
- Viewport-aware j/k scrolling with g/G for top/bottom
- Enter opens a full-screen detail popup showing endpoints, binaries,
rationale, security notes, and action hints
Add approve/reject/approve-all draft actions:
- [a] approve selected chunk, [x] reject, [A] approve all pending
- Actions work from both the list view and the detail popup
- gRPC calls run async; result updates status bar and refreshes data
- Nav bar shows all available keybindings
Fix draft count refresh: sandbox_draft_counts now refreshes on every
tick (not just Dashboard), so the detail header badge updates in
real time.
Improve badge labels: show 'N pending' instead of a bare number in
both the dashboard sandbox list and sandbox detail header.
* refactor(policy): DB-level draft chunk dedup with hit counter and timestamps
Replace the in-memory HashSet dedup in SubmitPolicyAnalysis with a
database-level upsert. New denormalized columns on draft_policy_chunks:
- host, port: extracted from proposed_rule at insert time
- hit_count: incremented on conflict (same sandbox + host + port)
- first_seen_ms, last_seen_ms: track when the endpoint was first and
most recently proposed
A partial unique index (WHERE status IN ('pending','approved')) ensures
only one active chunk per endpoint per sandbox; rejected/superseded
chunks don't block new proposals.
Surface hit_count and first/last_seen in:
- CLI: 'openshell draft get' shows 'Hits: N (first ..., last ...)'
- TUI: detail popup shows hits row; list view shows 'Nx' suffix
* fix(policy): optimistic retry on policy version conflicts + structured logging
merge_chunk_into_policy and remove_chunk_from_policy now retry up to 5
times on UNIQUE constraint violations (version conflicts from concurrent
approvals). Each attempt re-reads the latest policy, re-merges the rule,
and increments the version. This eliminates the race condition where
rapid successive approvals would fail with a DB error.
Add structured tracing to all draft action handlers:
- ApproveDraftChunk: logs rule_name, host, port, hit_count before merge
and version + policy_hash after success
- RejectDraftChunk: logs rule_name, host, port, reason
- ApproveAllDraftChunks: logs pending_count at start, per-chunk merge
progress, and final summary with chunks_approved/skipped
- UndoDraftChunk: logs before/after with rule_name and version
- Retry attempts log as warnings with attempt number and conflicting
version
* wip: forward proxy fix, mapper allowed_ips, TUI polish, CTF rewrite
* fix(tui): use correct --gateway flag for ssh-proxy ProxyCommand
* chore: add Docker cleanup script for stale images, volumes, and build cache
* feat(tui): approve-all confirmation modal and CTF cleanup
Add [A] confirmation popup that snapshots pending chunks, shows a
scrollable list, and approves each chunk individually on confirm.
This prevents approving chunks that arrived after the modal opened.
Remove transient issue #205 reference from CTF victory banner.
* fix(tui): correct import ordering for rustfmt
* wip: stateful toggle model, rename to network rules
Draft chunks now follow a toggle state machine:
pending -> approved | rejected (initial decision)
approved <-> rejected (toggle)
One row per (sandbox_id, host, port) via expanded unique index.
Rejecting an approved rule removes it from the active policy.
Re-approving a rejected rule merges it back.
Rename CLI from 'draft' to 'rule', TUI from 'Draft Recommendations'
to 'Network Rules'. State-aware keybindings: approved shows [x] Revoke,
rejected shows [a] Approve. Fix sandbox detail hiding delete confirmation
behind pending message.
* refactor(policy): move mapper sandbox-side, slim schema, per-binary granularity
Move mechanistic mapper from gateway to sandbox so all analysis runs
sandbox-side (N sandboxes = N independent pipelines). Gateway is now a
thin validate + persist + approval layer.
Architectural changes:
- Move mechanistic_mapper.rs from navigator-server to navigator-sandbox
- Sandbox flush flow: aggregator drains -> mapper runs -> proposals sent
- Gateway SubmitPolicyAnalysis: validate + persist only, no mapper
- Drop denial_summaries table (write-only, zero readers)
- Consolidate migrations 003+004+005 into single 003
Schema slimming:
- Drop 5 unused columns from draft_policy_chunks (stage, denial_refs,
supersedes_chunk_id, analysis_mode, decided_by)
- Add per-binary granularity: binary column, widen unique index to
(sandbox_id, host, port, binary)
- Mapper groups by (host, port, binary), one proposal per triple
- Merge appends binary to existing rule; revoke removes just that binary
CTF & UX:
- 7-gate CTF: add Gate 3 (curl -> ifconfig.me:80) for per-binary demo
- TUI shows binary short name in list, full path in detail popup
- CLI output shows binary field
- Idempotent rule names, hit_count accumulates real denial counts
- Rationale text no longer bakes in stale denial count
* fix(cli): improve completion coverage and gateway selection
Closes#232
Add typed completion metadata for gateway, sandbox, and path arguments, make gateway selection interactive on TTYs, and expose root subcommands in completions while keeping non-interactive behavior stable.
Add --gpu flag to 'openshell gateway start' that enables NVIDIA GPU
access inside the cluster container. When enabled, the Docker container
receives GPU device requests via bollard, k3s auto-detects the NVIDIA
runtime, and the NVIDIA k8s-device-plugin is deployed via HelmChart CR
so Kubernetes workloads can request nvidia.com/gpu resources.
- Add GPU device request to Docker container creation (bollard DeviceRequest)
- Set GPU_ENABLED=true env var to gate GPU manifest deployment
- Multi-stage Dockerfile build to bundle NVIDIA Container Toolkit binaries
- Conditional GPU manifest copy in cluster entrypoint
- HelmChart CR for nvidia-device-plugin v0.18.2 with GFD and NFD enabled
- Thread gpu bool through DeployOptions, CLI flag, and run dispatch
---------
Co-authored-by: Piotr Mlocek <pmlocek@nvidia.com>
* refactor(docker): rename server image to gateway
Rename Dockerfile.server to Dockerfile.gateway and update all image
references from openshell/server to openshell/gateway across Helm
charts, Kubernetes manifests, mise tasks, build/deploy scripts, CI
workflows, and documentation.
The underlying Rust binary (navigator-server) is unchanged -- this
rename only affects the Docker image name and Dockerfile.
* fix: catch remaining server->gateway references in docs and comments
* feat(inference): add sandbox-system inference route for platform-level inference
Add a separate system-level inference route that the sandbox supervisor
can use in-process for platform functions (e.g., embedded agent harness
for policy analysis), distinct from the user-facing inference.local
endpoint. The system route is accessed via an in-process API on the
supervisor, ensuring userland code in the sandbox netns cannot reach it.
- Extend proto with route_name fields on Set/Get inference messages
- Add ResolvedRoute.name field to the router for route segregation
- Server resolves both user and sandbox-system routes in bundles
- Sandbox partitions routes into user/system caches on refresh
- Expose InferenceContext::system_inference() in-process API
- CLI --sandbox flag targets the system route on set/get/update
- Integration tests using mock:// routes for the full in-process path
Closes#207
* refactor(cli): rename --sandbox flag to --system for inference commands
The --sandbox flag could be misread as targeting a user-level sandbox
operation. Rename to --system to clearly indicate it configures the
platform-level system inference route.
* style(cli): collapse short if-else per rustfmt 1.94
---------
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
Closes#196
Piggyback on the existing 2-second sandbox list refresh to detect policy
version changes. When the user is on the sandbox screen and the listed
current_policy_version differs from the displayed version, fetch the
updated policy via GetSandboxPolicy. Scroll position is preserved across
auto-refreshes so the user is not disrupted.
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
Drew Newberry