trending/OpenShell
1
0
Fork 0
mirror of https://github.com/NVIDIA/OpenShell.git synced 2026-10-17 05:31:16 +08:00
Code Issues Packages Projects Releases Wiki Activity
1,455 Commits 487 Branches 151 Tags
vm-runtime
Commit Graph
31 Commits
Author SHA1 Message Date
krishicks cbf026366d fix(ocsf): require network activity endpoints (#3355) 
Previously, Network Activity could be constructed without a source or
destination endpoint, allowing connection, accept, relay, and configuration
events to violate the OCSF 1.8 endpoint constraint.

Now, NetworkActivityBuilder requires a source or destination endpoint at
compile time. Connection failures identify the workload peer or genuine
transparent destination, listener failures identify the listening endpoint,
and mediation-lane failures use Application Lifecycle rather than fabricated
network endpoints. Malformed forward requests use HTTP Activity with a
method-only request, generated 400 response, and workload peer.

Additionally, Unix relay-channel events use Base Event, policy-validation
warnings use Config State Change, and the unused bypass monitor is removed
because the current isolation architecture no longer uses it.

Signed-off-by: Kris Hicks <khicks@nvidia.com>
 2026-09-21 17:13:41 +00:00
krishicks 481ce566e1 fix(ocsf): correct HTTP activity context (#3316) 
Previously, metadata events omitted both HTTP request and response objects,
early proxy rejections used HTTP Activity without request context, and
unsupported-scheme events did not expose enough safe HTTP context to satisfy
the OCSF 1.8 schema.

Now, metadata events include a method-only request and their actual HTTP
response codes without recording the metadata URL. Unsupported-scheme events
also include a method-only request plus the generated 400 response. Authority
mismatches and credential-resolution denials use HTTP Activity with their
generated 403 or 500 responses, and HTTP activity IDs are derived from the
request method.

Additionally, HttpActivityBuilder now enforces the OCSF request-or-response
constraint at compile time.

Signed-off-by: Kris Hicks <khicks@nvidia.com>
 2026-09-15 18:47:55 +00:00
Piotr Mlocek 5b57f0d154 fix(ci): restore Windows test portability (#3288) 
* fix(ci): restore Windows test portability

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

* fix(tasks): skip Unix lockfile check on Windows

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

* fix(tasks): use buf shim on Windows

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

---------

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
 2026-09-11 21:40:08 +00:00
araza008Jamie KingDrew Newberry
 		ae57979b03 feat(mxc): add Windows ETW-to-OCSF audit trail (#3015) 
* feat(mxc): ETW->OCSF audit consumer + Windows OCSF JSONL parity (cp6 P1)

Add a Windows MXC ETW->OCSF audit trail in openshell-driver-mxc: a real-time
Sandboxing-provider ETW consumer that decodes events (TDH), attributes each to
an OpenShell sandbox_id, and maps them to OCSF (lifecycle 6002, config 5019,
process 1007, finding 2004).

cp6 Phase 1 - durable OCSF JSONL audit-file parity with Linux:
- openshell-ocsf: add emit_ocsf_event_routed (populates the event-bridge
  thread-local AND stamps sandbox_id+message in one dispatch) plus public
  set/clear_current_event; OS-aware device (Device::windows/for_current_os) so
  device.os.name reflects the host instead of a hardcoded Linux stub.
- etw_consumer: emit via the routed emit (previously fired a bare info! that
  never populated the bridge, so the structured event was dropped).
- openshell-server: install OcsfJsonlLayer over a synchronous daily-rotated
  appender (durable under force-kill), gated by OPENSHELL_OCSF_JSON, path via
  %PROGRAMDATA%\OpenShell\logs (override OPENSHELL_OCSF_LOG_DIR).
- device.hostname now resolves to the real gateway machine name.

Box-proven on 7F203-MXC-001: JSONL lines == shorthand OCSF rows, all valid
OCSF JSON, per-sandbox attribution intact, disabled state writes nothing.

Signed-off-by: Akber Raza <akberr@nvidia.com>

* feat(mxc): map remaining Sandboxing ETW events to OCSF

Close the last three ETW->OCSF gaps so the audit trail covers the full
set of events the Sandboxing provider emits (12/12):

- ProcessLaunched -> Process Activity [1007] "Launch" (confirmed start;
  carries the real processId/threadId, the twin of CreateProcessInSandbox
  which only has the request + command line).
- SandboxProxyConfigured -> Device Config State Change [5019] (the one
  network-plane setup event; surfaces proxyPort, "no proxy" when 0).
- SandboxConsoleReferencePlumbed -> Device Config State Change [5019]
  (console-handle plumbing).

map_config_state now handles the full config/hardening/setup family and
carries proxyPort/hasConsoleReference/creationFlags as unmapped fields.
Verified on 7F203-MXC-001: 11/12 event types emit OCSF without a proxy
(SandboxProxyConfigured requires proxy config to fire).

Signed-off-by: Akber Raza <akberr@nvidia.com>

* fix(mxc): seed ETW attribution under registry lock + Device tests

Address CodeRabbit review on !31:

- Prevent stale ETW attribution on a delete/launch race: register the
  wxc-exec pid while holding the registry lock, and bail if the sandbox
  entry is already gone. Previously the attribution key could be seeded
  after `delete` had removed the sandbox, leaving a stale key that could
  misroute later Sandboxing ETW events to a dead sandbox_id. Lock order
  (registry -> attribution) matches the delete path, so no deadlock.
- Add unit tests for the new Device::windows and Device::for_current_os
  constructors to harden Windows/Linux OCSF device parity.

Signed-off-by: Akber Raza <akberr@nvidia.com>

* fix(mxc-etw): buffer+replay racing events and harden attribution keys

Addresses two ETW->OCSF attribution review items (Shailendra #1, #2).

#2 early-event loss: ETW delivers the sandbox create/config burst the instant wxc-exec starts, which can beat the driver's register_launch (now under the registry lock post-Ready). process_event previously dropped anything unresolved, losing the racing burst. Add a bounded, time-bounded pending buffer (PENDING_MAX=4096, PENDING_TTL=5s): unresolved events are held and replayed once attribution lands, aged-out ones dropped. Consumer switched to a timed recv_timeout(200ms) so the buffer is re-driven after each event and on a tick. Emit path factored into shared emit_resolved().

#1 attribution collisions: a Windows PID is recycled after exit and a command line is commonly identical across sandboxes. register_launch now rebinds by_pid on reuse and clears the stale last_pid_sid hint (warns if the PID still pointed at a different, leaked sandbox); command line is held in by_cmd only while unique and demoted to a new ambiguous_cmds set on a second owner, so a duplicate command refuses to resolve rather than misroute.

Unit tests: buffer replay (direct + cross-link), buffer bound, PID-reuse rebind, duplicate-cmd non-resolution. Box-verified on 7F203-MXC-001 (5 sandboxes, identical cmd -> 5 isolated sandbox_ids, 50/50 OCSF/JSONL, BuffersLost=0).

Signed-off-by: Akber Raza <akberr@nvidia.com>

* docs(mxc-etw): note cmd_line is captured raw with no privacy filtering

Review item #3 (Shailendra): add a PRIVACY NOTE on map_process_launch stating cmd_line is copied verbatim into OCSF process.cmd_line with no redaction, so secrets/PII on a command line land unredacted in the durable audit trail (deliberate audit-fidelity trade-off; treat the log as sensitive). Redaction is owned by an upstream privacy layer, not this path; no general audit-output PII scrubber exists today (openshell_core::secrets [CREDENTIAL] redaction is scoped to the proxy HTTP-target logging, a separate egress path).

Signed-off-by: Akber Raza <akberr@nvidia.com>

* fix(mxc-etw): open ETW trace on caller thread so start_session reports real status

Review item #4 (Shailendra): start_session previously returned Ok(EtwSession) as soon as the pump thread was spawned, but OpenTraceW ran later inside that thread; if it failed we still handed back a live-looking session and logged 'consumer started' (silent failure = false audit coverage).

Split the two Win32 calls instead of adding a channel handshake (avoids any lost-wakeup/hang risk): the quick, synchronous OpenTraceW now runs on the caller thread (open_trace), and only the blocking ProcessTrace runs on the pump thread (run_trace). start_session returns Err if OpenTraceW fails (reclaiming the boxed Sender so the consumer disconnects, stopping the session, joining the consumer) and returns Ok/logs 'started' only once capture is genuinely open. Opened handle + LoggerName buffer + boxed Sender are carried to the pump via a Send OpenedTrace so they outlive ProcessTrace.

Box-verified on 7F203-MXC-001: consumer started=True, failed-to-start=False, 50 OCSF rows / 50 JSONL, BuffersLost=0 (no regression to capture/emit).

Signed-off-by: Akber Raza <akberr@nvidia.com>

* fix(mxc-etw): guard pending-event replay against PID recycling

CodeRabbit flagged that drain_resolved() re-resolved buffered events
against the live by_pid map, so if Windows recycled a wxc-exec PID within
PENDING_TTL a stale event from the dead sandbox could be emitted under the
new owner.

Stamp each by_pid registration with its Instant and add resolve_replay(),
used only on the buffered/replay path. It (a) never falls back to the
recycle-/ambiguity-prone by_cmd or last_pid_sid keys, and (b) trusts a PID
match only when the registration is not newer than the buffered event by
more than REPLAY_PID_GRACE (2s) - a recycled PID's registration lands well
outside that window, so the stale event ages out instead of misattributing.
The legitimate #2 seed race (registration lands ~immediately) still replays.

Adds unit tests for the recycle-refusal, in-grace acceptance, and
weak-fallback exclusion.

Signed-off-by: Akber Raza <akberr@nvidia.com>

* fix(mxc-etw): surface unexpected ProcessTrace termination (review #4)

start_session already returns Err on OpenTraceW failure (runs on the
caller thread since e41a7701), closing the first half of Shailendra's #4.
This closes the second half: ProcessTrace's result was discarded, so if
capture died mid-run the backend had no way to know.

Add a shared CaptureHealth (stopped/stopping/exit_code) between the pump
thread and EtwSession. run_trace now records ProcessTrace's WIN32_ERROR
and, when the pump returns without a deliberate stop, logs at ERROR that
MXC OCSF capture is no longer running. EtwSession::stop() sets `stopping`
before teardown so a normal shutdown isn't misreported, and
EtwSession::is_capture_alive() exposes the state for status/diagnostics.

Box-verified on 7F203-MXC-001: 5 sandboxes, 50 attributed OCSF rows,
JSONL parity 50/50, BuffersLost=0, clean start/stop (no false failure).

Signed-off-by: Akber Raza <akberr@nvidia.com>

* feat(mxc-ocsf): add ETW->OCSF audit-trail example kit; fix proxy-configured message

Add a runnable OCSF audit-trail example under examples/ (run-ocsf-audit.ps1,
mxc-ocsf-audit.toml, ocsf-audit.yaml, README) that spins up sandboxes with the
in-process ETW consumer and egress proxy on, emitting a full OCSF JSONL audit
trail across all four classes (6002/5019/1007/2004).

Fix SandboxProxyConfigured mapping to log "MXC sandbox proxy configured" instead
of a misleading "(no proxy)" when the provider reports proxyPort=0; the event's
presence already indicates proxy configuration. Verified on-box: 26 events, all
mapped ETW event types present.

Signed-off-by: Akber Raza <akberr@nvidia.com>

* feat(mxc-ocsf): clearer audit report + client-safe run-ocsf-audit.ps1

Improve the ETW to OCSF audit-trail example output and make it safe to ship.

Report:
- Add an event-type coverage count ("N of M expected event types fired");
  the denominator auto-adjusts (8 with proxy on, 7 with -NoProxy).
- Split the checklist into expected event types vs anomaly findings
  (ActivityError/FallbackError), which are reported separately and not
  counted toward coverage (a clean run may emit none).
- Verdict is now coverage-based (all expected types must fire) instead of
  the looser "at least 3 OCSF classes".
- Call out the absolute path to the durable OCSF JSONL log prominently.

Client-safety:
- Default -ShareOut to empty (no auto-copy); pass -ShareOut a UNC path to
  opt in. Removes a hardcoded internal share path from a published example.
- Drop internal-team wording ("Hand that zip back for evaluation", "BUNDLE:")
  in favor of neutral "Results bundle:".
- Update README-ocsf-audit.txt to match the opt-in -ShareOut behavior.

Verified on both MXC boxes: 7F203-MXC-001 (base-container) -> PASS, 8 of 8
event types, 26 OCSF events across 4 classes; 7F203-MXC-003 (AppContainer
fallback) -> reduced set as expected, clean output.

Signed-off-by: Akber Raza <akberr@nvidia.com>

* fix(mxc): configure OCSF audit workloads per sandbox

- remove unsupported gateway-scoped workload fields from the shipped MXC audit example.
- build the command, working directory, and filesystem grant from each run's ShareDir
- pass the workload through --driver-config-json.
- preserve the host CONNECT proxy configuration and conditional audit coverage for the future host_connect_proxy merge
- require the workload output when determining the audit verdict.

Signed-off-by: Akber Raza <akberr@nvidia.com>

* fix(mxc): omit command arguments from OCSF audit logs

- record only the executable basename for MXC CreateProcessInSandbox audit events
- leave process.cmd_line unset so workload arguments cannot reach shorthand or JSONL logs
- cover tokens, passwords, signed URLs, and PII with a secret-leak regression test
- update the audit example, architecture guidance, and published logging documentation
- preserve ETW attribution and future host_connect_proxy enforcement behavior

Signed-off-by: Akber Raza <akberr@nvidia.com>

* feat(etw): enhance ETW session management with distinct naming for concurrent gateways

* fix(etw): bound the audit queue during overload

- replace the unbounded ETW callback channel with count- and byte-bounded buffering

- keep the ETW callback non-blocking and count records rejected during overload

- emit immediate, rate-limited warnings that identify resulting audit coverage gaps

- make the audit example fail when queue overload causes dropped ETW records

- cover stalled consumers, oversized events, and warning throttling with unit tests

Signed-off-by: Akber Raza <akberr@nvidia.com>

* fix(etw): harden sandbox audit attribution

- remove command-line and persistent per-PID fallback keys from live and replay resolution

- retire the driver-owned wxc-exec PID before publishing child completion

- retain established identity, activity, and correlation-vector links only for the five-second late-event window

- prevent buffered records from crossing rapid PID retirement and reuse boundaries

- add resolver and lifecycle coverage and document the attribution trust boundary

Signed-off-by: Akber Raza <akberr@nvidia.com>

* chore(mxc): address rebase follow-ups

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(mxc): align OCSF audit example with driver config

- remove unsupported egress proxy settings

- stop requiring the unavailable proxy audit event

- update example documentation for supported event coverage

Signed-off-by: Akber Raza <akberr@nvidia.com>

* fix(etw): redact command-line secrets in DecodedEtwEvent summary

* fix(etw): enhance PID resolution and event attribution logic for ETW records

* fix(ocsf): restrict gateway-local JSONL sink to Windows/MXC path with opt-in configuration

* address rebase issues

* fix(mxc): address ETW audit review feedback

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(mxc): fail closed across ambiguous PID reuse

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(mxc): bind ETW attribution to process generation

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Akber Raza <akberr@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Jamie King <jamiek@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
 2026-09-11 02:04:26 +00:00
krishicks 3c0f58872e refactor(ocsf): rename SandboxContext to EventContext (#3263) 
Rename the shared OCSF context and its callers without changing event behavior.

Signed-off-by: Kris Hicks <khicks@nvidia.com>
 2026-09-10 21:46:08 +00:00
krishicks 67374efdf8 fix(ocsf): emit schema-valid event identities (#3247) 
Previously, every event from a sandbox reused the sandbox ID as its event
ID. Consumers deduplicating security records could mistake separate events
for the same record, and missing device types or empty image objects could
prevent schema validation.

Give each event its own ID, retain the sandbox association separately, and
classify the environment as Other/Sandbox while keeping the OS separate.
Omit unknown container details instead of emitting empty objects.

Security tooling can now distinguish events from the same sandbox and read
their identity consistently after serialization.

Refs #1055

Signed-off-by: Kris Hicks <khicks@nvidia.com>
 2026-09-10 16:35:24 +00:00
John T. Myers f4dc6be4b2 refactor(inference): remove managed inference routes (#3195) 
* refactor(inference): remove managed inference routes

Closes #3172

Remove the inference route control plane, inference.local data path, built-in router crate, and SDK surface. Move inference workloads to explicitly imported provider profiles and native endpoints, with migration cleanup and updated tests and documentation.

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(policy): preserve alternate upstream isolation

Restore the provider policy activation guard so legacy OpenAI and Anthropic providers configured for alternate base URLs do not grant egress to the built-in public vendor endpoints.

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

---------

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
 2026-09-09 18:47:22 +00:00
Adel ZaaloukandJohn Myers 48a8a4bf09 feat(ocsf): configurable schema version for SIEM backward compatibility (#2717) 
* feat(ocsf): configurable schema version for SIEM backward compatibility

Add a gateway-configurable OCSF schema version target that downgrades
JSONL output for SIEMs that only support older schema versions. AWS
Security Lake requires v1.1.0, Splunk CIM Add-On targets v1.1-v1.3.

The downgrade filter strips profile-gated fields (ai_model, container,
observation_point_id), removes unknown profiles from metadata.profiles,
rewrites metadata.version, and adds an unmapped.downgraded_from
breadcrumb so auditors can distinguish "no model involved" from "model
attribution stripped."

Supported target versions (1.1, 1.3) are enforced by an allow-list in
the settings registry. Invalid values are rejected with a clear error.

The setting flows to sandboxes via the settings bundle and takes effect
on the next poll cycle. The shorthand log output is unaffected.

Closes #2662

Signed-off-by: Adel Zaalouk <zanetworker@gmail.com>
Signed-off-by: Adel Zaalouk <azaalouk@redhat.com>

* fix(ocsf): align downgrade with schema version

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

---------

Signed-off-by: Adel Zaalouk <zanetworker@gmail.com>
Signed-off-by: Adel Zaalouk <azaalouk@redhat.com>
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com>
 2026-09-04 18:43:58 +00:00
Yuedong Wu c93b2fa7da docs(gateway-config): fix stale community sandbox image path (#2800) 
* docs(gateway-config): fix stale community sandbox image path

Signed-off-by: Yuedong Wu <dwcn22@outlook.com>

* docs(sandbox-image): purge remaining stale image references

Rebasing onto main surfaced four more instances of the same dead
ghcr.io/nvidia/openshell/sandbox path, introduced by commits merged
after this branch was opened: three test fixtures (driver-docker,
openshell-ocsf, compute::mod) and one user-facing default in the
SPIFFE token-exchange Podman demo README. Correct all four to
ghcr.io/nvidia/openshell-community/sandboxes/base, consistent with
the rest of this fix.

Signed-off-by: Yuedong Wu <dwcn22@outlook.com>

---------

Signed-off-by: Yuedong Wu <dwcn22@outlook.com>
 2026-09-04 00:16:42 +00:00
krishicks 22073fcaae test: backfill coverage for OCSF logging (#3065) 
Characterization tests for behavior that later work changes. No behavior
change.

- Pin the `openshell logs` output format, so a change to how that text is
  produced shows up as a diff rather than passing silently.
- Cover the supervisor log push layer.
- Assert OcsfEvent survives a JSON round trip. Serialization is written
  by hand and deserialization dispatches on class_uid into independent
  per-variant paths, so a field can serialize correctly and still be
  dropped or rejected on the way back in.

Refs #1055

Signed-off-by: Kris Hicks <khicks@nvidia.com>
 2026-08-31 23:02:30 +00:00
krishicks 4eaa1051f8 docs: correct some comments and references (#3059) 
OCSF_VERSION has been 1.8.0; two descriptions still said 1.7.0.

The default_gateway_id doc comment described a hostname fallback the
implementation never had.  A per-replica default would be wrong here:
gateway_id is embedded in the JWT iss/aud, so it has to be stable across
replicas.

Signed-off-by: Kris Hicks <khicks@nvidia.com>
 2026-08-31 18:39:53 +00:00
John T. Myers b2ea81822b feat(network): enable Docker and Podman policy DNS and transparent TCP (#2723) 
* feat(network): enable Docker transparent TCP egress

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(e2e): cover Docker transparent TCP egress

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* feat(network): correlate transparent TCP audit events

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* docs(examples): add transparent TCP Redis demo

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* docs(examples): demonstrate blocked TCP connections

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* docs(examples): focus Redis demo audit output

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(network): close transparent TCP policy bypasses

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(sandbox): reject unsupported TCP policy reloads

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(ci): satisfy Linux transparent TCP lints

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* feat(podman): enable transparent TCP egress

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(podman): permit policy DNS port binding

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(e2e): use qualified transparent TCP hostname

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(podman): preserve exact policy DNS names

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(podman): route policy DNS over TCP

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(dns): serve multiple TCP queries per connection

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* docs(network): explain native DNS and TCP egress

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(sandbox): reconcile runtime reload with upstream

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(network): harden transparent DNS capture

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(podman): preserve resolver behavior for native tcp

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* docs(network): clarify native tcp runtime constraints

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(network): remove unused transparent tcp pin

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(network): admit redirected transparent tcp

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(network): restore podman transparent networking

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(podman): permit alpine busybox binaries

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(podman): use portable alpine keepalive

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(podman): build musl networking fixture

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(podman): isolate musl DNS probe

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(podman): keep privileged port capability dropped

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(network): preserve transparent TCP port 53

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(network): report synthetic pool pressure by family

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(podman): bind tcp fixtures before readiness

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(podman): grant fixture low-port bind

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

---------

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
 2026-08-20 19:26:53 +00:00
Simon Scatton 9505ca5ed1 chore: remove Bazel build support (#2840) 
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
 2026-08-20 15:08:09 +00:00
Adel ZaaloukandClaude Opus 4.6 600bbae845 feat(ocsf): emit AI inference events via ai_operation profile on ApiActivity (#2664) 
Apply the official OCSF ai_operation profile (introduced in v1.8.0) to
ApiActivity [6003] events when the inference proxy routes a model call
through inference.local. Attaches an ai_model object (name, ai_provider)
and puts token counts and latency in unmapped fields.

ApiActivity [6003] is the schema-correct class for the ai_operation
profile in v1.8.0 (HttpActivity only gets it in v1.9.0). In Splunk CIM,
ApiActivity maps to the "Change" data model, naturally separating
inference events from regular HTTP proxy traffic.

Changes:
- Add AiModel object and ai_model field on BaseEventData
- Add ApiActivityEvent struct and ApiActivityBuilder
- Add emit_ai_inference in proxy.rs using ApiActivity with ai_operation
- Vendor OCSF v1.8.0 schemas including api_activity class, ai_model
  object, and ai_operation profile definitions
- Bump OCSF_VERSION to 1.8.0
- Update schema validation to skip profile-gated required fields

Shorthand: API:INFERENCE [INFO] claude-3-haiku via anthropic 701ms [POST /v1/messages]

Splunk/SIEM backward compatibility (v1.1/v1.3 CIM mapping) is tracked
separately in #2662 as a configurable serialization concern.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-08-18 16:30:03 +00:00
Simon Scatton 0a3ec7a112 feat(bazel): add rustfmt checks to tests (#2599) 2026-08-03 16:37:15 +00:00
Simon Scatton 1959ea19be build(bazel): establish RFC 0012 Rust reference graph (#2414) 
Establish the Phase 1 reference implementation proposed by RFC 0012
while retaining the existing Cargo and Mise workflows during evaluation.

- pin Bazel 9.1.1 and configure Bzlmod, rules_rs, LLVM, protobuf, and
  Rust 1.95 toolchains
- import third-party crates from Cargo metadata and propagate the
  workspace version into Bazel targets
- add library, binary, proc-macro, unit-test, and integration-test
  targets across the supported Rust workspace crates and drivers
- generate protobuf Rust sources and descriptor sets under Bazel while
  preserving Cargo-compatible generated-code imports
- annotate aws-lc-sys and zstd-sys native dependencies, build Z3 4.15.2
  from source, and generate z3-sys bindings
- make CLI and procfs test fixtures available as explicit Bazel inputs
  without relying on fixed host binary paths
- define optimized release targets for Linux x86_64 and aarch64 CLI,
  sandbox, and gateway binaries, plus macOS aarch64 artifacts
- add Bazel, buildifier, and lcov to the Nix development environment

RFC: 0012 (rfc12 branch)
Refs: #2491

Signed-off-by: Simon Scatton <sscatton@nvidia.com>
 2026-08-03 09:05:40 +00:00
Piotr Mlocek d556748771 feat(supervisor-middleware): add network egress middleware (#2027) 
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
 2026-07-16 17:47:49 -07:00
Kirit Thadaka ccdac9cec4 fix(mcp): include tool names in policy logs (#2189) 
Signed-off-by: Kirit93 <kthadaka@nvidia.com>
 2026-07-10 08:38:26 -07:00
rhubenovandEvan Lezar 1ca23bc5c8 refactor(openshell-sandbox): Split sandbox into process and network subcrates. (#1650) 
* refactor(sandbox): extract run_networking from run_sandbox

Lifts TLS state generation, network namespace setup, proxy startup,
bypass monitor spawn, and SSH-side proxy URL / netns FD computation
out of run_sandbox into a sibling async fn `run_networking` that
returns a Networking struct. The identity cache moves with it (only
consumed by the proxy). Entrypoint PID allocation moves just above
the call site so it can be passed in.

No behavior changes — same OCSF emits, same async order, same RAII
lifetimes for the proxy and bypass-monitor handles, now held by the
returned Networking value in run_sandbox's frame.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(sandbox): extract run_process and lift netns to run_sandbox

Lifts the post-networking tail of `run_sandbox` (zombie reaper, SSH
server, supervisor session, process spawn, OPA probe, policy poll loop,
denial aggregator, wait/exit) into a sibling async fn `run_process`.

Also moves network namespace creation out of `run_networking` into a new
`create_netns_for_proxy` helper invoked from `run_sandbox`, so
`run_networking` is purely the proxy component (OPA evaluation, TLS
interception, credential injection, inference routing, gRPC control
API). The netns is then borrowed into both `run_networking` and
`run_process`.

No behavior change.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* chore(workspace): scaffold openshell-supervisor-networking and openshell-supervisor-process crates

Add empty placeholder crates that subsequent commits will populate as the
sandbox decomposition proceeds. Both crates compile clean as part of the
workspace and are picked up automatically by the existing
`members = ["crates/*"]` glob.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(core): lift DenialEvent to openshell-core

The DenialEvent struct is emitted by both the proxy/L7 layer (networking-side)
and the bypass monitor (process-side), and crosses the run_networking ->
run_process API boundary. Move it to openshell-core so the eventual
supervisor-networking and supervisor-process crates can both reference it
without depending on each other. DenialAggregator and the channel/flush
helpers stay in openshell-sandbox for now.

A thin `pub use openshell_core::DenialEvent;` re-export from
denial_aggregator.rs keeps every existing `crate::denial_aggregator::DenialEvent`
call site resolving without further edits.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(core): lift normalize_path to openshell-core

Move the lexical path-normalization helper from openshell-policy to
openshell-core::paths so it can be reached from crates that sit below
openshell-policy in the dependency graph. openshell-policy keeps its
existing public API via a `pub use` re-export, so all current call sites
(e.g. openshell-sandbox/src/policy.rs) continue to resolve unchanged.

This is a prerequisite for lifting openshell-sandbox/src/policy.rs into
openshell-core: that file's `From<ProtoFilesystemPolicy>` impl calls
normalize_path, and lifting it as-is would cycle through openshell-policy.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(core): lift SandboxPolicy and friends to openshell-core

Move openshell-sandbox/src/policy.rs (SandboxPolicy, NetworkPolicy,
ProxyPolicy, FilesystemPolicy, LandlockPolicy, ProcessPolicy, NetworkMode,
LandlockCompatibility, plus their Proto* TryFrom/From impls) to
openshell-core/src/policy.rs.

Both prospective supervisor leaves (networking and process) dispatch on
SandboxPolicy. Hosting it in openshell-core lets either leaf reach for it
without depending on the other (or on the future orchestrator).

The From<ProtoFilesystemPolicy> impl now calls the in-crate
openshell_core::paths::normalize_path lifted in the previous commit, which
is what made this move cycle-free.

Update all crate::policy::* call sites in openshell-sandbox to
openshell_core::policy::*.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-process): move child_env from openshell-sandbox

child_env (proxy_env_vars, tls_env_vars) is process-side only — consumed
by process.rs and ssh.rs. With the orchestrator staying in
openshell-sandbox (Shape A), openshell-sandbox depends on the new leaf
crates, so process-only modules can land in
openshell-supervisor-process directly.

Add openshell-supervisor-process as a path dependency of
openshell-sandbox. Update process.rs and ssh.rs to import from
openshell_supervisor_process::child_env.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-process): move skills from openshell-sandbox

Move the static skills installer (and its embedded resource directory)
out of openshell-sandbox into openshell-supervisor-process. The module
is process-side only — invoked once during sandbox start to drop
agent skill files into the workspace — and has no cross-leaf consumers.

Adds miette as a dependency and tempfile as a dev-dependency on
openshell-supervisor-process.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-networking): move mechanistic_mapper from openshell-sandbox

Move the mechanistic mapper (HTTP method/path → operation classifier
that derives policy proposals from connection summaries) out of
openshell-sandbox into openshell-supervisor-networking. Single internal
caller (run_policy_poll_loop in lib.rs) and only depends on
openshell-core + tracing — no cross-leaf entanglement.

First population of the openshell-supervisor-networking crate; adds
openshell-core and tracing as dependencies.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(core): lift procfs to openshell-core

Move procfs (PID lookups, ancestor walking, /proc/net/tcp socket-owner
resolution, file SHA256 hashing) from openshell-sandbox into
openshell-core. The module is consumed cross-leaf — by bypass_monitor
on the process side and by identity / proxy on the networking side —
so it has to sit below both leaves.

Adds tracing, sha2, and hex as dependencies on openshell-core.
Updates the three call sites in openshell-sandbox to import from
openshell_core::procfs.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-networking): move identity from openshell-sandbox

Move BinaryIdentityCache (path → SHA256 cache used to identify the
process behind an outbound connection) from openshell-sandbox into
openshell-supervisor-networking. The cache is consumed only by the
networking-side proxy and the orchestrator; with procfs already in
openshell-core there are no remaining cross-leaf dependencies.

Adds miette as a dependency and tempfile as a dev-dependency on
openshell-supervisor-networking. Adds a Default impl for
BinaryIdentityCache to satisfy clippy::new_without_default now that
the type is publicly exposed across crates.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-process): move agent-proposals flag from openshell-sandbox

Move AGENT_PROPOSALS_ENABLED, agent_proposals_enabled(), and the
test-only ProposalsFlagGuard out of openshell-sandbox into
openshell-supervisor-process::proposals. The flag is read only by the
process-side policy_local route handler and the orchestrator; lifting
it to openshell-core would have made core carry sandbox-owned runtime
state without buying anything.

The test-only ProposalsFlagGuard is still consumed from networking-side
l7/rest tests today (until the wider Q2 OCSF/gRPC injection work lands).
Expose it via a new optional `test-helpers` feature on
openshell-supervisor-process so test crates opt in explicitly without
pulling tokio sync primitives into production builds.

openshell-sandbox keeps its existing crate-private path
(`crate::AGENT_PROPOSALS_ENABLED`, `crate::test_helpers`) via re-exports
so call sites and tests are unchanged.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(core): lift secrets to openshell-core

Move crates/openshell-sandbox/src/secrets.rs to crates/openshell-core/src/secrets.rs so both supervisor leaves can reach SecretResolver and the placeholder helpers without depending on openshell-sandbox.

Add base64 to openshell-core deps (only stdlib + base64 are used). Promote previously pub(crate) constructors and methods on SecretResolver to pub since cross-crate callers (provider_credentials, proxy/L7 tests) now name them across the crate boundary. Update import paths in proxy.rs, l7/{rest,relay,websocket}.rs, and provider_credentials.rs from crate::secrets to openshell_core::secrets.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(core): lift provider_credentials to openshell-core

Move crates/openshell-sandbox/src/provider_credentials.rs to crates/openshell-core/src/provider_credentials.rs. Both supervisor leaves now name ProviderCredentialState in their function signatures (run_networking takes &ProviderCredentialState, run_process takes ProviderCredentialState by value), and under Shape A leaves can't depend on openshell-sandbox, so the type must live in openshell-core.

The orchestrator (run_sandbox in openshell-sandbox) remains the only writer: it constructs ProviderCredentialState::from_environment and the policy poll loop calls install_environment on credential rotation. Both leaves stay pure readers via snapshot()/resolver().

Update import paths in proxy.rs, ssh.rs, and lib.rs from crate::provider_credentials to openshell_core::provider_credentials.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* style: rustfmt import ordering

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(ocsf): move SandboxContext singleton from openshell-sandbox

Move the process-wide OCSF SandboxContext OnceLock + LazyLock fallback + getter from openshell-sandbox/src/lib.rs into a new openshell-ocsf::ctx module. The type already lives in openshell-ocsf, so its singleton lives next to it.

Add openshell_ocsf::ctx::set_ctx() and openshell_ocsf::ctx::ctx(). The orchestrator (run_sandbox) now calls set_ctx during startup. Sandbox keeps a pub(crate) use openshell_ocsf::ctx::ctx as ocsf_ctx; re-export so the 138 existing crate::ocsf_ctx() call sites resolve unchanged.

When the sandbox modules themselves migrate into the leaf crates, they'll import openshell_ocsf::ctx directly and the re-export goes away.

Under Shape A neither leaf can depend on openshell-sandbox; both already depend on openshell-ocsf to construct events, so this adds no new dep edge.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(core): lift grpc_client to openshell-core

Both prospective leaves (supervisor-networking and supervisor-process)
need CachedOpenShellClient, AuthedChannel, and the connect/fetch
helpers. Under Shape A the leaves cannot depend on openshell-sandbox,
so the type has to live below them. openshell-core already pulls in
tonic and miette; this enables tonic's channel/tls features and adds
tokio as a direct dep.

Updates all crate::grpc_client::* call sites in openshell-sandbox to
openshell_core::grpc_client::*. No re-export shim — the call-site
count was small enough to update directly.

See architecture/plans/sandbox-split-design-choices.md for the full
rationale and trade-offs.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-networking): move denial_aggregator from openshell-sandbox

DenialAggregator and FlushableDenialSummary belong with the proxy and L7
layer that emit denials. Moves the file into openshell-supervisor-networking;
adds tokio as a regular dep there since DenialAggregator uses
tokio::sync::mpsc.

Drops the pub use openshell_core::DenialEvent re-export inside the moved
file (no longer needed cross-crate). Updates bypass_monitor.rs, proxy.rs,
and lib.rs to import openshell_core::DenialEvent directly.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-process): move log_push from openshell-sandbox

LogPushLayer is a process-side tracing layer that streams sandbox logs
to the gateway via gRPC. Moves into openshell-supervisor-process; adds
openshell-core, openshell-ocsf, tokio-stream, tracing, and
tracing-subscriber as direct deps there.

Updates the only external call site (openshell-sandbox/src/main.rs) to
import from openshell_supervisor_process::log_push.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-process): move bypass_monitor from openshell-sandbox

bypass_monitor reads /dev/kmsg for nftables drop log lines and emits
denial events. Pure process-side concern, called only from
run_networking which spawns it on the netns. Moves into
openshell-supervisor-process; all deps (openshell-core, openshell-ocsf,
tokio, tracing) were already declared there.

Replaces crate::ocsf_ctx() shim calls inside the moved file with
openshell_ocsf::ctx::ctx() — first leaf-side caller to import the OCSF
context singleton directly instead of going through openshell-sandbox's
re-export.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-process): move debug_rpc from openshell-sandbox

debug_rpc is the CLI subcommand handler that exercises authenticated
gRPC calls (issue-token, refresh-token, get-config, etc.). Pure
process-side concern, called only from openshell-sandbox/main.rs.

Adds base64, hex, serde_json, sha2, and tonic (with channel/tls
features) as direct deps on openshell-supervisor-process. Updates the
single call site in main.rs.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-process): move supervisor_session from openshell-sandbox

supervisor_session opens a bidirectional gRPC stream that lets the
gateway initiate shells inside the sandbox. Pure process-side concern,
called only from run_process. Adds uuid as a direct dep on
openshell-supervisor-process.

Replaces crate::ocsf_ctx() shim calls inside the moved file with
openshell_ocsf::ctx::ctx() — same pattern as bypass_monitor.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-process): lift managed_children tracker from openshell-sandbox

The MANAGED_CHILDREN set tracks PIDs of supervisor-spawned children
(entrypoint + SSH sessions) so the orchestrator's SIGCHLD reaper can
distinguish them from incidental zombies. Pure process-side concern,
moves to openshell_supervisor_process::managed_children with three
public fns: register, unregister, is_managed.

Updates lib.rs reaper, process.rs, and ssh.rs to call through the new
module path. Drops the now-unused HashSet import from lib.rs.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-process): move sandbox hardening from openshell-sandbox

Lift the process-only hardening pieces (landlock, seccomp, PreparedSandbox,
prepare/enforce, log_sandbox_readiness, top-level apply, and
apply_supervisor_startup_hardening) from crates/openshell-sandbox/src/sandbox/
to crates/openshell-supervisor-process/src/sandbox/.

Leave netns.rs and nft_ruleset.rs in openshell-sandbox for now, since both
eventual leaf crates (supervisor-networking and supervisor-process) read from
NetworkNamespace and its final home is decided when run_networking and
run_process are extracted.

Replace crate::ocsf_ctx() shims in landlock.rs and the new linux/mod.rs with
direct openshell_ocsf::ctx::ctx() calls. Update call sites in lib.rs,
process.rs, and ssh.rs to import sandbox from openshell_supervisor_process
while keeping the netns import unchanged.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(core): lift proposals flag from openshell-supervisor-process

Move proposals.rs (AGENT_PROPOSALS_ENABLED OnceLock + agent_proposals_enabled
reader + test_helpers::ProposalsFlagGuard) from openshell-supervisor-process
to openshell-core so both eventual leaf crates can read it without depending
on each other.

The flag is process-wide singleton state initialised once during sandbox
startup and read by both the policy.local route (networking-side) and the
skills installer (process-side) — same shape as openshell_ocsf::ctx.

Move the test-helpers Cargo feature alongside it: openshell-core gains the
feature, openshell-supervisor-process loses it, and openshell-sandbox's
dev-dependency now enables openshell-core/test-helpers. Update the sandbox
re-export shim to point at openshell_core::proposals.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(core): lift netns + nft_ruleset from openshell-sandbox

Move NetworkNamespace and the nft_ruleset bypass-rule generator from
crates/openshell-sandbox/src/sandbox/linux/ to crates/openshell-core/src/netns/.
Both eventual leaf crates (supervisor-networking and supervisor-process) read
from NetworkNamespace, so it must live somewhere both can depend on without
violating the Shape A no-leaf-to-leaf rule.

Replace crate::ocsf_ctx() shims in netns with direct openshell_ocsf::ctx::ctx()
calls, matching the pattern used in already-migrated process modules. Update
super::nft_ruleset references inside netns to nft_ruleset since the module
is now a sibling sub-module of netns/mod.rs.

Add openshell-ocsf and uuid as linux-only dependencies of openshell-core, and
gate pub mod netns on target_os = "linux" since the implementation uses
netlink, ip(8), and namespace fds. Delete the now-empty sandbox/{mod.rs,
linux/mod.rs} stubs and update NetworkNamespace import paths in lib.rs and
process.rs to point at openshell_core::netns.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-process): move process.rs and ssh.rs from openshell-sandbox

Lift the entrypoint process spawn module and the embedded SSH server
module into openshell-supervisor-process. openshell-sandbox now
re-exports ProcessHandle/ProcessStatus and calls
openshell_supervisor_process::ssh::run_ssh_server directly.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-networking): move proxy, l7, opa, policy_local from openshell-sandbox

Lift the egress proxy, L7 enforcement modules, OPA engine, and policy.local
advisor API into openshell-supervisor-networking. Move accompanying data
files (sandbox-policy.rego), test fixtures (testdata/), and integration
tests (system_inference, websocket_upgrade). Sandbox lib.rs now references
these via openshell_supervisor_networking::* and ProxyHandle::start_with_bind_addr
is exposed as pub for the orchestrator call site.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(sandbox): hoist policy poll loop and denial aggregator into orchestrator

Move the symlink-resolver, policy poll loop, and denial-aggregator flush
spawns out of run_process and into run_sandbox so run_process no longer
needs OpaEngine, retained_proto, the local policy context, the sandbox
name, the gateway endpoint for telemetry, the OCSF flag, or the denial
receiver. These long-running orchestrator-owned tasks now live alongside
the other sandbox-startup wiring, matching the design log decision in
architecture/plans/sandbox-split-design-choices.md (Q5).

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-process): move run_process from openshell-sandbox

Lift the workload supervision entry point (zombie reaper, SSH server
spawn, supervisor session, entrypoint child spawn, exit-with-timeout)
into its own module in openshell-supervisor-process. The orchestrator
in openshell-sandbox now calls openshell_supervisor_process::run::run_process
directly. With this move run_process names only types from openshell-core,
openshell-ocsf, openshell-supervisor-process itself, std, and tokio —
no openshell-supervisor-networking dependency.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-networking): move bypass_monitor from supervisor-process

Bypass detection is network-policy enforcement: it parses nftables LOG
entries from /dev/kmsg and emits OCSF NetworkActivity / DetectionFinding
events plus DenialEvents into the same channel the proxy feeds. Its
lifetime is tied to the network namespace, not to the workload child.
Moving it to openshell-supervisor-networking puts it next to the proxy
and the denial aggregator that consume its output, and unblocks moving
run_networking out of openshell-sandbox without a leaf-to-leaf dep.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-networking): move inference route helpers from openshell-sandbox

Move build_inference_context, partition_routes, bundle_to_resolved_routes,
spawn_route_refresh, the InferenceRouteSource enum, and the route refresh
interval helpers into a new openshell-supervisor-networking::inference_routes
module along with their unit tests. The orchestrator now calls into the
networking leaf for inference context construction; the leaf owns its own
route bundle resolution end-to-end.

The new module is named inference_routes to avoid colliding with the
existing l7::inference module, which handles request-time HTTP parsing
and pattern matching rather than route bundle setup.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-networking): move run_networking from openshell-sandbox

Move the Networking handle struct, run_networking, and the Linux-only
create_netns_for_proxy helper into a new openshell-supervisor-networking::run
module. The orchestrator in openshell-sandbox now invokes
openshell_supervisor_networking::run::{create_netns_for_proxy, run_networking}
and reads the Networking fields directly; the leaf owns the entire
networking-stack startup path (CA generation, proxy task, bypass monitor,
inference context, denial channel) end-to-end.

The Networking RAII handle fields (proxy, bypass_monitor) are now public
without leading underscores so the public API satisfies clippy's
pub_underscore_fields lint while still serving as drop guards held by the
orchestrator's frame.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* fix(workspace): align Cargo deps and call sites for split crates

The recent module lifts left two Linux-only gaps that the macOS host
workspace check skipped:

- openshell-core's netns module needs libc, tempfile, and nix on Linux,
  but only openshell-ocsf and uuid were carried over.
- openshell-supervisor-process's seccomp/landlock modules need landlock
  and seccompiler, which still lived on openshell-sandbox.
- openshell-sandbox's runtime_pid_limit branch referenced an unqualified
  process:: that pointed at the old in-crate module.

Move landlock/seccompiler to supervisor-process, add the missing core
deps, qualify the call sites, and drop sandbox deps that no longer have
runtime users (landlock, seccompiler, target-gated tempfile/uuid, the
unix libc/rustix block).

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-network): rename openshell-supervisor-networking to openshell-supervisor-network

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-network): own denial-aggregator flush end-to-end

Move the denial-aggregator spawn and flush_proposals_to_gateway out of
run_sandbox and into run_networking. The networking leaf already owns
every other input (proxy + bypass_monitor as producers, denial channel,
mechanistic_mapper, denial_aggregator) and already opens its own gRPC
connections (inference_routes, policy_local) — the orchestrator was the
only piece left straddling the boundary.

Networking now drives the full path: producers -> channel -> aggregator
-> flush -> gateway. Drops denial_rx from Networking; adds sandbox_name
to run_networking so SubmitPolicyAnalysis can resolve by sandbox name
(falls back to ID when unset). Same shape as log_push in the process leaf.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-network): own symlink-resolution task

Move the OPA binary-symlink resolver out of run_sandbox and into
run_networking. The task probes /proc/<entrypoint_pid>/root/ until the
workload's mount namespace is accessible, then rebuilds the OPA engine
with resolved binary paths so policy rules match canonical names instead
of symlinks.

Both inputs (Arc<OpaEngine>, retained_proto) are networking-leaf concerns
and were already plumbed into run_networking; the entrypoint_pid Arc is
read lazily after the process leaf populates it. Adds retained_proto as
a parameter and spawns the resolver early in run_networking so the probe
loop starts before the proxy comes up.

Same shape as the denial-flush move: networking owns its own background
task end-to-end; the orchestrator stops hosting work that doesn't
conceptually belong to it.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-process): move seccomp install into run_process

The supervisor seccomp prelude is part of "set up the workload-side
process tree", not part of orchestration. Move the call site from
run_sandbox into the top of run_process and drop the now-unused
re-export from openshell-sandbox::lib.

Timing is preserved: by the time the orchestrator calls run_process,
run_networking has already returned, so netns + nftables setup is
complete.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-process): move check_runtime_pid_limit into run_process

The PID-limit precondition is process-side: it gates whether the workload
child can be spawned at all. Move the call from run_sandbox into the top
of run_process, alongside the seccomp prelude. Same shape as the seccomp
move — function already lives process-side, only the call site relocates.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-process): move validate_sandbox_user to process crate

The sandbox-user check is a precondition for privilege-dropping the
workload child; it has no relevance to networking. Move the function
next to drop_privileges in openshell-supervisor-process::process and
call it from the top of run_process.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-process): move prepare_filesystem to process crate

Creating and chowning read_write directories is workload-side
preparation, not orchestration. Move prepare_filesystem and its
prepare_read_write_path helper (plus tests) into
openshell-supervisor-process::process and call from run_process,
alongside validate_sandbox_user.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-process): move startup skill install into run_process

The eager initial-settings fetch + agent skill install is process-side:
the install materializes files the workload's filesystem sees. The
orchestrator still owns the AGENT_PROPOSALS_ENABLED OnceLock init
because the policy poll loop also reads it; only the early fetch and
install hop into run_process.

Behavior unchanged. Best-effort: any RPC or install failure is logged
but does not fail startup.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-network): own PolicyLocalContext construction

Move the PolicyLocalContext construction from run_sandbox into
run_networking. The orchestrator was building it solely to thread it into
the networking leaf and to share it with the policy poll loop; now
run_networking builds it from inputs it already takes (retained_proto,
openshell_endpoint, sandbox_name|sandbox_id) and exposes it on the
returned Networking struct.

The orchestrator's poll loop now grabs the Arc clone from
networking.policy_local_ctx, so the orchestrator no longer imports
openshell_supervisor_network::policy_local.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* feat(supervisor): add --mode flag to gate network/process leaves

Add a --mode flag (default "network,process") that selects which
supervisor leaves run in the current process. Two new shapes are
unlocked without splitting the binary:

  --mode=network             # network-only sidecar
  --mode=process             # process-only supervisor
  --mode=network,process     # combined (default; current behavior)

In network-only mode the orchestrator skips run_process and waits on
SIGINT/SIGTERM before tearing down the proxy. The entrypoint PID stays
at 0 for the lifetime of the process, which silently degrades the
proxy's binary-identity TOFU and the bypass monitor's PID enrichment;
this is correct in a split-pod topology where the workload's /proc
lives in another pod.

In process-only mode run_networking is skipped entirely. SSH sessions
get no proxy URL, no netns FD, and no CA paths, matching what a
split-pod consumer would expect when network enforcement is delegated
to a sidecar.

The policy poll loop continues to run unconditionally; its OPA-reload
and policy.local hooks already gate on the resources only present when
network is enabled, and the env-refresh / proposals-toggle hooks
remain active in process mode.

Closes a step toward the RFC-0001 supervisor topology proposed in
issue #1305 by drew.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* style(supervisor-process): rustfmt long debug! line

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-network): pull DenialEvent down from core

DenialEvent is only emitted and consumed inside openshell-supervisor-network
(proxy, bypass monitor, denial aggregator). It never crossed the leaf
boundary, so the earlier lift to openshell-core was speculative. Move it
back into the network crate where its only callers live.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-network): pull procfs down from core

procfs was lifted to openshell-core under the assumption it would be
shared cross-leaf, but on the current branch all three callers
(bypass_monitor, identity, proxy) live in openshell-supervisor-network.
No file in openshell-supervisor-process imports it. Move the module to
the network crate and drop sha2/hex from openshell-core, which were
pulled in only for procfs.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* style(supervisor-network): run cargo fmt

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* fix(supervisor-network): add libc dev-dependency for procfs tests

The procfs/bypass_monitor/proxy test modules use libc::{fork, exec,
fcntl, kill, waitpid} but the dep wasn't declared in this crate's
Cargo.toml. It was previously satisfied transitively when these
modules lived in openshell-core; the move left the test target
unable to resolve libc.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(sandbox): move denial aggregator to orchestrator

The denial aggregator and mechanistic mapper consume denial events
produced by the proxy and (subsequently) the bypass monitor. With both
supervisor leaves becoming pure producers of `DenialEvent`, the
consumer-side aggregation belongs in the orchestrator, not in either
leaf.

Move `denial_aggregator.rs` and `mechanistic_mapper.rs` from
`openshell-supervisor-network` to `openshell-sandbox` (the
orchestrator). The orchestrator now owns the unbounded denial channel:
it constructs `(tx, rx)`, hands `tx` to `run_networking` for the proxy
to clone, drains `rx` via the aggregator task, and runs the gateway
flush helper itself.

`run_networking`'s signature gains a `denial_tx` parameter and loses
its internal channel construction, aggregator spawn, and
`flush_proposals_to_gateway` helper. `DenialEvent` stays in
`openshell-supervisor-network` for now; a follow-up commit will lift
it to `openshell-core` alongside the bypass monitor relocation.

* refactor(supervisor-process): pull bypass monitor down from network

`bypass_monitor` is process-isolation machinery: it tails the kernel
log via `dmesg --follow`, parses nftables LOG lines emitted from the
workload's network namespace, resolves PIDs via `/proc`, and emits
OCSF events plus optional `DenialEvent`s. None of this touches the
proxy, OPA, TLS, or any other supervisor-network state — it only
shared the denial channel because both feed the same aggregator.

Move `bypass_monitor.rs` from `openshell-supervisor-network` to
`openshell-supervisor-process` (as `bypass_monitor/mod.rs`). Spawn it
in `run_process` where the netns name and entrypoint PID are already
in scope. The orchestrator hands an extra `bypass_denial_tx` clone of
the denial channel sender to `run_process` for this purpose.

Lift `DenialEvent` from `openshell-supervisor-network` to
`openshell-core`. Both supervisor leaves now produce it, so it needs
a shared location that neither leaf depends on. This reverses an
earlier commit that pulled the type into the network leaf when it was
the only producer.

Copy the minimal subset of `/proc` parsers used by `bypass_monitor`
into a private `bypass_monitor::procfs` submodule. The alternative —
extracting a shared procfs crate — is a much larger refactor that
this commit does not need; supervisor-network's `procfs.rs` continues
to serve the proxy and identity cache.

* refactor(supervisor-process): derive ssh netns fd inside run_process

The ssh_netns_fd was computed in run_networking purely to forward it
through the Networking struct and back into run_process. supervisor-network
never read it. Move the derivation to run_process where the
NetworkNamespace handle is already in scope.

* refactor(supervisor-process): derive ssh proxy url inside run_process

The ssh_proxy_url was computed in run_networking purely to forward it
through the Networking struct and back into run_process. supervisor-network
never read it. Move the derivation to run_process where the
NetworkNamespace handle and SandboxPolicy are already in scope.

After this commit the Networking struct no longer carries any SSH-shaped
fields, and supervisor-network reads only host_ip from the netns (for the
proxy bind address).

* refactor(supervisor-network): take proxy bind ip directly instead of netns

run_networking only ever read host_ip from the netns it was passed (the
SSH plumbing reads moved to run_process in earlier commits). Replace the
NetworkNamespace parameter with a plain Option<IpAddr> the orchestrator
extracts. supervisor-network's run module no longer references the netns
type for any consumer, only for create_netns_for_proxy (which still lives
in this crate; relocates next).

* refactor(supervisor-process): move netns ownership out of core

Relocates the NetworkNamespace handle, nft ruleset builder, and
create_netns_for_proxy constructor into openshell-supervisor-process.
The orchestrator (openshell-sandbox) phantom-owns the RAII handle for
the duration of run_sandbox; supervisor-network no longer references
the type at all.

Drops uuid, libc, nix, openshell-ocsf, and tempfile from core's Linux
target deps (all were exclusive to netns). tempfile becomes a Linux
runtime dep on supervisor-process for nft ruleset application.

* chore(sandbox): prune leaf-only deps from orchestrator manifest

cargo-machete flagged 26 direct dependencies that were carried over
from the pre-split monolith and are no longer used by the orchestrator
itself: regorus, russh, rcgen, tokio-rustls, ipnet, apollo-parser,
openshell-router, anyhow, base64, bytes, flate2, glob, hex, hmac, nix,
rand_core, rustls-pemfile, serde, serde_yml, sha1, sha2, thiserror,
tokio-stream, uuid, webpki-roots.

These now live (transitively) in openshell-supervisor-network and
openshell-supervisor-process where they are actually consumed.

* chore(deps): prune unused deps from supervisor crates

- Drop unused `url` from openshell-supervisor-network.
- Mark `prost` and `prost-types` as cargo-machete-ignored in
  openshell-core: they have no source-level `use`, but the tonic-
  generated proto code references them via `::prost::Message` etc.
- openshell-supervisor-process is already clean.

* fix(supervisor-network): wait for entrypoint PID before symlink probe

The OPA symlink-resolution task reads entrypoint_pid once at the top
of the spawned closure. Because the spawn happens before run_process
publishes the workload PID, the load returns 0, the probe path bakes
in as /proc/0/root/, and the loop exhausts its retries against a path
that does not exist on Linux. The reload never fires, so policies that
whitelist symlinked binaries (e.g. /usr/bin/python3 → python3.11) get
silent denials when the workload exec's the realpath.

Split the wait into two phases: 5s polling entrypoint_pid for a
non-zero value, then the existing 5s window probing /proc/<pid>/root/.
Distinct warn messages on each timeout so future debugging can tell
"PID never published" apart from "container fs never appeared".

* fix(sandbox): restore GPU procfs baseline (#1522)

Signed-off-by: Evan Lezar <elezar@nvidia.com>
(cherry picked from commit 5102cb9413)
Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* fix(supervisor-process): use renamed tonic tls-native-roots feature

Upstream renamed the tonic `tls` feature to `tls-native-roots`. The
supervisor-process Cargo.toml still referenced the old name, which broke
the workspace build after merging upstream.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* refactor(supervisor-network): relocate token_grant and spiffe_endpoint

Upstream's SPIFFE-backed token grant feature landed in
crates/openshell-sandbox/src/. After the supervisor split, the L7
enforcement code in supervisor-network calls into token_grant, which
would require supervisor-network to depend back on sandbox.

Move token_grant.rs and spiffe_endpoint.rs into supervisor-network
where the only callers live, add the reqwest and spiffe deps to
supervisor-network's Cargo.toml, and drop them from sandbox.

Also fix two stale `openshell_core::proto::` self-references in
openshell-core (a pre-existing breakage that surfaced once the rest of
the merge compiled).

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

* fix(supervisor-process): broaden Path import cfg to all unix targets

The `Path` import was gated on `cfg(any(test, target_os = "linux"))`,
but `prepare_read_write_path` is gated on `cfg(unix)` — broader. On
non-Linux unix the function still referenced `&std::path::Path`
explicitly, so upstream's qualified path was load-bearing.

After the supervisor split, lint runs on Linux where `Path` IS in
scope, so `unused_qualifications` fires. Broaden the import cfg to
match the function's cfg and use the bare `Path` name everywhere.

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>

---------

Signed-off-by: Radoslav Hubenov <rrhubenov@gmail.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>
 2026-06-15 15:16:23 -04:00
LateNightHackathonandlatenighthackathon 84c24a0e6d fix(ocsf): widen the shorthand [reason:] budget so denial endpoints stay readable (NVIDIA/NemoClaw#4760) (#1799) 
A denied egress logs an OCSF NET:OPEN DENIED line whose [reason:] field carries the full destination endpoint and the rejecting policy name, but the shorthand formatter capped the reason at 80 bytes and appended a literal "...". A typical reason is longer than that, so the shorthand denial logs showed the endpoint cut mid-string with the port and policy name lost:

  NET:OPEN [MED] DENIED ... [reason:endpoint host.example:44...]

Raise MAX_REASON_LEN to 256 so a full endpoint plus policy name fits, and route both reason_tag and message_tag through a shared char-boundary truncation helper. The previous `&text[..MAX]` byte slice would also panic if the cut fell inside a multibyte UTF-8 character.

Per review: reason_tag now also normalizes embedded newlines to spaces before truncating (matching message_tag), so endpoint- or error-derived text cannot inject CR/LF into the single-line shorthand (CWE-117); the multibyte-boundary test now slices inside a multibyte character to actually exercise the step-back.

The issue was reported on the NemoClaw tracker (NVIDIA/NemoClaw#4760), but the OCSF shorthand emitter is OpenShell code, so the fix lands here. Cross-repo references do not auto-close, so no closing keyword is used.

Ref: NVIDIA/NemoClaw#4760  (https://github.com/NVIDIA/NemoClaw/issues/4760)

Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
Co-authored-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
 2026-06-10 09:36:31 -07:00
Eric Curtin 1399f371e0 refactor: deduplicate OCSF builder setters and persistence helpers (#1800) 
Eliminate three categories of repeated code:

1. OCSF builder setter methods
   Add six macros to builders/mod.rs (impl_activity_setter!,
   impl_action_disposition_setters!, impl_actor_process_setter!,
   impl_dst_endpoint_setter!, impl_src_endpoint_addr_setter!,
   impl_firewall_rule_setter!) that generate the identical one-liner
   setters duplicated across network, http, ssh, process, finding,
   and lifecycle builders. Removes ~55 hand-written method bodies.

2. Persistence protobuf decode-and-hydrate block
   Extract a private decode_record<T>() helper that decodes an
   ObjectRecord into a protobuf message and hydrates resource_version.
   The identical three-line block was repeated in get_message,
   get_message_by_name, list_messages, and list_messages_with_selector.
   Each method now delegates to decode_record via .map().transpose()
   or .into_iter().map().collect().

3. Store enum dispatch arms
   Add a store_dispatch! macro that expands
   match self { Postgres(s) => s.method(...).await, Sqlite(s) => ... }.
   Applied to all 13 dispatch-only methods on Store, removing ~80
   lines of mechanical match arms with zero logic.
 2026-06-08 08:59:29 -07:00
Alexander Watson e98ea3ee93 feat(policy): add agentic approval loop (#1528) 2026-05-29 21:11:31 -07:00
Eric Curtin 9857fa1974 refactor: deduplicate shared code across ocsf builders and driver crates (#1526) 
Extract repeated patterns into shared helpers:

- Add impl_builder_setters! macro to openshell-ocsf/builders that
  generates the identical severity(), status(), and message() setter
  methods present on all 7 OCSF event builders
- Add SandboxContext::apply_common_fields() to consolidate the
  four-line build() finalization (set_status, set_message, set_device,
  set_container) repeated in every builder
- Add driver_utils::sandbox_token_path() to centralize the XDG state
  path construction for sandbox JWT files used by both the Docker and
  Podman drivers
- Add driver_utils::build_capabilities_response() to eliminate the
  identical GetCapabilitiesResponse struct literal repeated across the
  Docker, Podman, and Kubernetes compute drivers
 2026-05-26 08:12:43 -07:00
Russell Bryant c5d1d76d94 refactor(sandbox): replace iptables with nftables for network policy enforcement (#1401) 
Migrate all sandbox and VM driver network policy enforcement from
iptables to nftables. nftables provides atomic ruleset loading, a
cleaner rule syntax, and is the standard netfilter interface in modern
kernels.

Sandbox bypass enforcement (openshell-sandbox):
- Replace iptables chain of individual rule insertions with a single
  atomic nftables ruleset load via nft -f
- New nft_ruleset module with pure functions for ruleset generation
  and unit tests
- Combine log and reject rules in one inet family table (handles both
  IPv4 and IPv6 in a single ruleset)
- Fall back to reject-only ruleset when kernel lacks nft_log support
- Enable net.netfilter.nf_log_all_netns so log rules work from
  non-init network namespaces
- Use temp file for nft ruleset loading instead of stdin for
  compatibility with minimal VM guest environments

VM TAP networking (openshell-driver-vm):
- Replace iptables NAT/forwarding rules with nftables equivalents
- New nft_ruleset module for TAP network rule generation with unit
  tests
- Atomic table-per-TAP-device lifecycle (create/destroy)
- Host-side rules provide NAT infrastructure and defense-in-depth
  isolation (input chain restricts VM to gateway port only, forward
  chain blocks unsolicited inbound); primary security enforcement
  happens inside the VM guest via the sandbox supervisor's own rules

VM init script:
- Load nft kernel modules at sandbox init
- Enable nf_log_all_netns sysctl for bypass detection logging

OCSF / docs:
- Update firewall rule engine references from iptables to nftables
- Document host firewall interaction model and two-layer enforcement
  architecture in VM driver README and compute drivers reference

Closes #1335

Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2026-05-18 16:37:17 -07:00
Piotr Mlocek 5abc36c461 feat(relay): route forwarding through ForwardTcp (#1029) 2026-05-11 21:23:37 -07:00
Drew Newberry 24724742a8 ci(rust): enforce -D warnings on clippy (#1008) 2026-04-29 12:12:53 -07:00
John T. Myers 1cabd2563c fix(sandbox): harden seccomp denylist, SSRF protection, and inference policy enforcement (#819) 
* fix(sandbox): harden seccomp denylist, SSRF protection, and inference policy enforcement

- Remove seccomp skip in NetworkMode::Allow so baseline syscall
  restrictions apply regardless of network mode
- Block cross-process manipulation syscalls (process_vm_writev,
  pidfd_open, pidfd_getfd, pidfd_send_signal) symmetric with existing
  ptrace and process_vm_readv blocks
- Block clone/clone3 with CLONE_NEWUSER flag, new mount API syscalls
  (fsopen, fsconfig, fsmount, fspick, move_mount, open_tree), and
  namespace manipulation (setns, umount2, pivot_root)
- Block userfaultfd and perf_event_open consistent with Docker default
  seccomp profile
- Deny and close keep-alive inference connections after a non-inference
  request instead of silently continuing the loop
- Add CGNAT (100.64.0.0/10), benchmarking (198.18.0.0/15), and other
  special-use IP ranges to SSRF protection in both proxy and
  mechanistic mapper

* fix(sandbox): install clone3 seccomp filter before main filter

* test(ocsf): fix shorthand firewall engine expectation
 2026-04-13 12:16:10 -07:00
John T. Myers 09af1b6bad fix(sandbox): add JSON bodies to proxy 403/502 responses and include port in HTTP log URLs (#809) 
L4 CONNECT and forward-proxy denial responses (403) and upstream
failure responses (502) were returned with empty bodies, making them
indistinguishable from each other when using curl -s. Add a
build_json_error_response() helper that produces consistent JSON
bodies with error and detail fields, matching the format already
used by the L7 deny path.

Also fix Url::to_display_string() in openshell-ocsf to include
non-default ports in the shorthand log output. Requests to e.g.
http://host:9876/path were logged as http://host/path, dropping the
port number.

Closes #807, closes #808
 2026-04-13 10:56:03 -07:00
John T. Myers 2ca553a4a0 fix(sandbox): validate always-blocked IPs at load time, enrich denial logs, and filter un-fixable proposals (#814) (#815) 
Policies with allowed_ips entries targeting loopback, link-local, or
unspecified ranges now fail at connection time instead of being silently
blocked at runtime. The shorthand log format for DENIED events includes
a [reason:...] suffix so operators can distinguish 'allowlist miss' from
'structurally un-allowable'. The mechanistic mapper skips proposals for
always-blocked destinations, preventing the infinite TUI notification
loop. The gateway validates proposed rules on approval as defense-in-depth.

- Extract shared IP helpers (is_always_blocked_ip, is_always_blocked_net,
  is_internal_ip) to openshell_core::net
- Reject always-blocked entries in parse_allowed_ips with hard error
- Skip implicit allowed_ips synthesis for always-blocked literal IP hosts
- Add status_detail to HttpActivityBuilder for denial reason propagation
- Enrich NET and HTTP shorthand with [reason:...] for DENIED events
- Add engine: tag to HTTP shorthand (consistency with NET shorthand)
- Filter always-blocked proposals in mechanistic mapper generate_proposals
- Add validate_rule_not_always_blocked server-side defense-in-depth
- Update architecture docs, published docs, and E2E test assertions
 2026-04-13 09:47:50 -07:00
John T. Myers b7779bdefa feat(sandbox): integrate OCSF structured logging for sandbox events (#720) 
* feat(sandbox): integrate OCSF structured logging for all sandbox events

WIP: Replace ad-hoc tracing calls with OCSF event builders across all
sandbox subsystems (network, SSH, process, filesystem, config, lifecycle).

- Register ocsf_logging_enabled setting (defaults false)
- Replace stdout/file fmt layers with OcsfShorthandLayer
- Add conditional OcsfJsonlLayer for /var/log/openshell-ocsf.log
- Update LogPushLayer to extract OCSF shorthand for gRPC push
- Migrate ~106 log sites to OCSF builders (NetworkActivity, HttpActivity,
  SshActivity, ProcessActivity, DetectionFinding, ConfigStateChange,
  AppLifecycle)
- Add openshell-ocsf to all Docker build contexts

* fix(scripts): attach provider to all smoke test phases to avoid rate limits

GitHub's unauthenticated API rate limit (60/hour) causes flaky 403s for
Phases 1, 2, and 4. Fix by attaching the provider to all sandboxes and
upgrading the Phase 1 policy to L7 so credential injection works.

Phase 4 (tls:skip) cannot inject credentials by design, so relax the
assertion to accept either 200 or 403 from upstream -- both prove the
proxy forwarded the request.

* fix(ocsf): remove timestamp from shorthand format to avoid double-timestamp

The display layer (gateway logs, TUI, sandbox logs CLI) already prepends
a timestamp. Having one in the shorthand output too produces redundant
double-timestamps like:

  15:49:11 sandbox INFO  15:49:11.649 I NET:OPEN ALLOWED ...

Now the shorthand is just the severity + structured content:

  15:49:11 sandbox INFO  I NET:OPEN ALLOWED ...

* refactor(ocsf): replace single-char severity with bracketed labels

Replace cryptic single-character severity codes (I/L/M/H/C/F) with
readable bracketed labels: [LOW], [MED], [HIGH], [CRIT], [FATAL].

Informational severity (the happy-path default) is omitted entirely to
keep normal log output clean and avoid redundancy with the tracing-level
INFO that the display layer already provides.

Before: sandbox INFO  I NET:OPEN ALLOWED ...
After:  sandbox INFO  NET:OPEN ALLOWED ...

Before: sandbox INFO  M NET:OPEN DENIED ...
After:  sandbox INFO  [MED] NET:OPEN DENIED ...

* feat(sandbox): use OCSF level label for structured events in log push

Set the level field to 'OCSF' instead of 'INFO' for OCSF events in the
gRPC log push. This visually distinguishes structured OCSF events from
plain tracing output in the TUI and CLI sandbox logs:

  sandbox OCSF  NET:OPEN [INFO] ALLOWED python3(42) -> api.example.com:443
  sandbox OCSF  NET:OPEN [MED] DENIED python3(42) -> blocked.com:443
  sandbox INFO  Fetching sandbox policy via gRPC

* fix(sandbox): convert new Landlock path-skip warning to OCSF

PR #677 added a warn!() for inaccessible Landlock paths in best-effort
mode. Convert to ConfigStateChangeBuilder with degraded state so it
flows through the OCSF shorthand format consistently.

* fix(sandbox): use rolling appender for OCSF JSONL file

Match the main openshell.log rotation mechanics (daily, 3 files max)
instead of a single unbounded append-only file. Prevents disk exhaustion
when ocsf_logging_enabled is left on in long-running sandboxes.

* fix(sandbox): address reviewer warnings for OCSF integration

W1: Remove redundant 'OCSF' prefix from shorthand file layer — the
    class name (NET:OPEN, HTTP:GET) already identifies structured events
    and the LogPushLayer separately sets the level field.

W2: Log a debug message when OCSF_CTX.set() is called a second time
    instead of silently discarding via let _.

W3: Document the boundary between OCSF-migrated events and intentionally
    plain tracing calls (DEBUG/TRACE, transient, internal plumbing).

W4: Migrate remaining iptables LOG rule failure warnings in netns.rs
    (IPv4 TCP/UDP, IPv6 TCP/UDP) to ConfigStateChangeBuilder for
    consistency with the IPv4 bypass rule failure already migrated.

W5: Migrate malformed inference request warn to NetworkActivity with
    ActivityId::Refuse and SeverityId::Medium.

W6: Use Medium severity for L7 deny decisions (both CONNECT tunnel and
    FORWARD proxy paths) to match the CONNECT deny severity pattern.
    Allows and audits remain Informational.

* refactor(sandbox): rename ocsf_logging_enabled to ocsf_json_enabled

The shorthand logs are already OCSF-structured events. The setting
specifically controls the JSONL file export, so the name should reflect
that: ocsf_json_enabled.

* fix(ocsf): add timestamps to shorthand file layer output

The OcsfShorthandLayer writes directly to the log file with no outer
display layer to supply timestamps. Add a UTC timestamp prefix to every
line so the file output matches what tracing::fmt used to provide.

Before: CONFIG:VALIDATED [INFO] Validated 'sandbox' user exists in image
After:  2026-04-01T15:49:11.649Z CONFIG:VALIDATED [INFO] Validated ...

* fix(docker): touch openshell-ocsf source to invalidate cargo cache

The supervisor-workspace stage touches sandbox and core sources to force
recompilation over the rust-deps dummy stubs, but openshell-ocsf was
missing. This caused the Docker cargo cache to use stale ocsf objects
from the deps stage, preventing changes to the ocsf crate (like the
timestamp fix) from appearing in the final binary.

Also adds a shorthand layer test verifying timestamp output, and drafts
the observability docs section.

* fix(ocsf): add OCSF level prefix to file layer shorthand output

Without a level prefix, OCSF events in the log file have no visual
anchor at the position where standard tracing lines show INFO/WARN.
This makes scanning the file harder since the eye has nothing consistent
to lock onto after the timestamp.

Before: 2026-04-01T04:04:13.065Z CONFIG:DISCOVERY [INFO] ...
After:  2026-04-01T04:04:13.065Z OCSF CONFIG:DISCOVERY [INFO] ...

* fix(ocsf): clean up shorthand formatting for listen and SSH events

- Fix double space in NET:LISTEN, SSH:LISTEN, and other events where
  action is empty (e.g., 'NET:LISTEN [INFO]  10.200.0.1' -> 'NET:LISTEN [INFO] 10.200.0.1')
- Add listen address to SSH:LISTEN event (was empty)
- Downgrade SSH handshake intermediate steps (reading preface, verifying)
  from OCSF events to debug!() traces. Only the final verdict
  (accepted/denied) is an OCSF event now, reducing noise from 3 events
  to 1 per SSH connection.
- Apply same spacing fix to HTTP shorthand for consistency.

* docs(observability): update examples with OCSF prefix and formatting fixes

Align doc examples with the deployed output:
- Add OCSF level prefix to all shorthand examples in the log file
- Show mixed OCSF + standard tracing in the file format section
- Update listen events (no double space, SSH includes address)
- Show one SSH:OPEN per connection instead of three
- Update grep patterns to use 'OCSF NET:' etc.

* docs(agents): add OCSF logging guidance to AGENTS.md

Add a Sandbox Logging (OCSF) section to AGENTS.md so agents have
in-context guidance for deciding whether new log emissions should use
OCSF structured logging or plain tracing. Covers event class selection,
severity guidelines, builder API usage, dual-emit pattern for security
findings, and the no-secrets rule.

Also adds openshell-ocsf to the Architecture Overview table.

* fix: remove workflow files accidentally included during rebase

These files were already merged to main in separate PRs. They got
pulled into our branch during rebase conflict resolution for the
deleted docs-preview-pr.yml file.

* docs(observability): use sandbox connect instead of raw SSH

Users access sandboxes via 'openshell sandbox connect', not direct SSH.

* fix(docs): correct settings CLI syntax in OCSF JSON export page

The settings CLI requires --key and --value named flags, not positional
arguments. Also fix the per-sandbox form: the sandbox name is a
positional argument, not a --sandbox flag.

* fix(e2e): update log assertions for OCSF shorthand format

The E2E tests asserted on the old tracing::fmt key=value format
(action=allow, l7_decision=audit, FORWARD, L7_REQUEST, always-blocked).
Update to match the new OCSF shorthand (ALLOWED/DENIED, HTTP:, NET:,
engine:ssrf, policy:).

* feat(sandbox): convert WebSocket upgrade log calls to OCSF

PR #718 added two log calls for WebSocket upgrade handling:

- 101 Switching Protocols info → NetworkActivity with Upgrade activity.
  This is a significant state change (L7 enforcement drops to raw relay).

- Unsolicited 101 without client Upgrade header → DetectionFinding with
  High severity. A non-compliant upstream sending 101 without a client
  Upgrade request could be attempting to bypass L7 inspection.
 2026-04-07 13:01:13 -07:00
John T. Myers 51aeffc9a7 feat(ocsf): create openshell-ocsf crate — standalone OCSF event types, formatters, and tracing layers (#489) 
feat(ocsf): create openshell-ocsf crate with OCSF v1.7.0 event types, formatters, and tracing layers
 2026-03-20 12:19:26 -07:00