mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-21 06:11:31 +08:00
* feat(mxc): ETW->OCSF audit consumer + Windows OCSF JSONL parity (cp6 P1) Add a Windows MXC ETW->OCSF audit trail in openshell-driver-mxc: a real-time Sandboxing-provider ETW consumer that decodes events (TDH), attributes each to an OpenShell sandbox_id, and maps them to OCSF (lifecycle 6002, config 5019, process 1007, finding 2004). cp6 Phase 1 - durable OCSF JSONL audit-file parity with Linux: - openshell-ocsf: add emit_ocsf_event_routed (populates the event-bridge thread-local AND stamps sandbox_id+message in one dispatch) plus public set/clear_current_event; OS-aware device (Device::windows/for_current_os) so device.os.name reflects the host instead of a hardcoded Linux stub. - etw_consumer: emit via the routed emit (previously fired a bare info! that never populated the bridge, so the structured event was dropped). - openshell-server: install OcsfJsonlLayer over a synchronous daily-rotated appender (durable under force-kill), gated by OPENSHELL_OCSF_JSON, path via %PROGRAMDATA%\OpenShell\logs (override OPENSHELL_OCSF_LOG_DIR). - device.hostname now resolves to the real gateway machine name. Box-proven on 7F203-MXC-001: JSONL lines == shorthand OCSF rows, all valid OCSF JSON, per-sandbox attribution intact, disabled state writes nothing. Signed-off-by: Akber Raza <akberr@nvidia.com> * feat(mxc): map remaining Sandboxing ETW events to OCSF Close the last three ETW->OCSF gaps so the audit trail covers the full set of events the Sandboxing provider emits (12/12): - ProcessLaunched -> Process Activity [1007] "Launch" (confirmed start; carries the real processId/threadId, the twin of CreateProcessInSandbox which only has the request + command line). - SandboxProxyConfigured -> Device Config State Change [5019] (the one network-plane setup event; surfaces proxyPort, "no proxy" when 0). - SandboxConsoleReferencePlumbed -> Device Config State Change [5019] (console-handle plumbing). map_config_state now handles the full config/hardening/setup family and carries proxyPort/hasConsoleReference/creationFlags as unmapped fields. Verified on 7F203-MXC-001: 11/12 event types emit OCSF without a proxy (SandboxProxyConfigured requires proxy config to fire). Signed-off-by: Akber Raza <akberr@nvidia.com> * fix(mxc): seed ETW attribution under registry lock + Device tests Address CodeRabbit review on !31: - Prevent stale ETW attribution on a delete/launch race: register the wxc-exec pid while holding the registry lock, and bail if the sandbox entry is already gone. Previously the attribution key could be seeded after `delete` had removed the sandbox, leaving a stale key that could misroute later Sandboxing ETW events to a dead sandbox_id. Lock order (registry -> attribution) matches the delete path, so no deadlock. - Add unit tests for the new Device::windows and Device::for_current_os constructors to harden Windows/Linux OCSF device parity. Signed-off-by: Akber Raza <akberr@nvidia.com> * fix(mxc-etw): buffer+replay racing events and harden attribution keys Addresses two ETW->OCSF attribution review items (Shailendra #1, #2). #2 early-event loss: ETW delivers the sandbox create/config burst the instant wxc-exec starts, which can beat the driver's register_launch (now under the registry lock post-Ready). process_event previously dropped anything unresolved, losing the racing burst. Add a bounded, time-bounded pending buffer (PENDING_MAX=4096, PENDING_TTL=5s): unresolved events are held and replayed once attribution lands, aged-out ones dropped. Consumer switched to a timed recv_timeout(200ms) so the buffer is re-driven after each event and on a tick. Emit path factored into shared emit_resolved(). #1 attribution collisions: a Windows PID is recycled after exit and a command line is commonly identical across sandboxes. register_launch now rebinds by_pid on reuse and clears the stale last_pid_sid hint (warns if the PID still pointed at a different, leaked sandbox); command line is held in by_cmd only while unique and demoted to a new ambiguous_cmds set on a second owner, so a duplicate command refuses to resolve rather than misroute. Unit tests: buffer replay (direct + cross-link), buffer bound, PID-reuse rebind, duplicate-cmd non-resolution. Box-verified on 7F203-MXC-001 (5 sandboxes, identical cmd -> 5 isolated sandbox_ids, 50/50 OCSF/JSONL, BuffersLost=0). Signed-off-by: Akber Raza <akberr@nvidia.com> * docs(mxc-etw): note cmd_line is captured raw with no privacy filtering Review item #3 (Shailendra): add a PRIVACY NOTE on map_process_launch stating cmd_line is copied verbatim into OCSF process.cmd_line with no redaction, so secrets/PII on a command line land unredacted in the durable audit trail (deliberate audit-fidelity trade-off; treat the log as sensitive). Redaction is owned by an upstream privacy layer, not this path; no general audit-output PII scrubber exists today (openshell_core::secrets [CREDENTIAL] redaction is scoped to the proxy HTTP-target logging, a separate egress path). Signed-off-by: Akber Raza <akberr@nvidia.com> * fix(mxc-etw): open ETW trace on caller thread so start_session reports real status Review item #4 (Shailendra): start_session previously returned Ok(EtwSession) as soon as the pump thread was spawned, but OpenTraceW ran later inside that thread; if it failed we still handed back a live-looking session and logged 'consumer started' (silent failure = false audit coverage). Split the two Win32 calls instead of adding a channel handshake (avoids any lost-wakeup/hang risk): the quick, synchronous OpenTraceW now runs on the caller thread (open_trace), and only the blocking ProcessTrace runs on the pump thread (run_trace). start_session returns Err if OpenTraceW fails (reclaiming the boxed Sender so the consumer disconnects, stopping the session, joining the consumer) and returns Ok/logs 'started' only once capture is genuinely open. Opened handle + LoggerName buffer + boxed Sender are carried to the pump via a Send OpenedTrace so they outlive ProcessTrace. Box-verified on 7F203-MXC-001: consumer started=True, failed-to-start=False, 50 OCSF rows / 50 JSONL, BuffersLost=0 (no regression to capture/emit). Signed-off-by: Akber Raza <akberr@nvidia.com> * fix(mxc-etw): guard pending-event replay against PID recycling CodeRabbit flagged that drain_resolved() re-resolved buffered events against the live by_pid map, so if Windows recycled a wxc-exec PID within PENDING_TTL a stale event from the dead sandbox could be emitted under the new owner. Stamp each by_pid registration with its Instant and add resolve_replay(), used only on the buffered/replay path. It (a) never falls back to the recycle-/ambiguity-prone by_cmd or last_pid_sid keys, and (b) trusts a PID match only when the registration is not newer than the buffered event by more than REPLAY_PID_GRACE (2s) - a recycled PID's registration lands well outside that window, so the stale event ages out instead of misattributing. The legitimate #2 seed race (registration lands ~immediately) still replays. Adds unit tests for the recycle-refusal, in-grace acceptance, and weak-fallback exclusion. Signed-off-by: Akber Raza <akberr@nvidia.com> * fix(mxc-etw): surface unexpected ProcessTrace termination (review #4) start_session already returns Err on OpenTraceW failure (runs on the caller thread since e41a7701), closing the first half of Shailendra's #4. This closes the second half: ProcessTrace's result was discarded, so if capture died mid-run the backend had no way to know. Add a shared CaptureHealth (stopped/stopping/exit_code) between the pump thread and EtwSession. run_trace now records ProcessTrace's WIN32_ERROR and, when the pump returns without a deliberate stop, logs at ERROR that MXC OCSF capture is no longer running. EtwSession::stop() sets `stopping` before teardown so a normal shutdown isn't misreported, and EtwSession::is_capture_alive() exposes the state for status/diagnostics. Box-verified on 7F203-MXC-001: 5 sandboxes, 50 attributed OCSF rows, JSONL parity 50/50, BuffersLost=0, clean start/stop (no false failure). Signed-off-by: Akber Raza <akberr@nvidia.com> * feat(mxc-ocsf): add ETW->OCSF audit-trail example kit; fix proxy-configured message Add a runnable OCSF audit-trail example under examples/ (run-ocsf-audit.ps1, mxc-ocsf-audit.toml, ocsf-audit.yaml, README) that spins up sandboxes with the in-process ETW consumer and egress proxy on, emitting a full OCSF JSONL audit trail across all four classes (6002/5019/1007/2004). Fix SandboxProxyConfigured mapping to log "MXC sandbox proxy configured" instead of a misleading "(no proxy)" when the provider reports proxyPort=0; the event's presence already indicates proxy configuration. Verified on-box: 26 events, all mapped ETW event types present. Signed-off-by: Akber Raza <akberr@nvidia.com> * feat(mxc-ocsf): clearer audit report + client-safe run-ocsf-audit.ps1 Improve the ETW to OCSF audit-trail example output and make it safe to ship. Report: - Add an event-type coverage count ("N of M expected event types fired"); the denominator auto-adjusts (8 with proxy on, 7 with -NoProxy). - Split the checklist into expected event types vs anomaly findings (ActivityError/FallbackError), which are reported separately and not counted toward coverage (a clean run may emit none). - Verdict is now coverage-based (all expected types must fire) instead of the looser "at least 3 OCSF classes". - Call out the absolute path to the durable OCSF JSONL log prominently. Client-safety: - Default -ShareOut to empty (no auto-copy); pass -ShareOut a UNC path to opt in. Removes a hardcoded internal share path from a published example. - Drop internal-team wording ("Hand that zip back for evaluation", "BUNDLE:") in favor of neutral "Results bundle:". - Update README-ocsf-audit.txt to match the opt-in -ShareOut behavior. Verified on both MXC boxes: 7F203-MXC-001 (base-container) -> PASS, 8 of 8 event types, 26 OCSF events across 4 classes; 7F203-MXC-003 (AppContainer fallback) -> reduced set as expected, clean output. Signed-off-by: Akber Raza <akberr@nvidia.com> * fix(mxc): configure OCSF audit workloads per sandbox - remove unsupported gateway-scoped workload fields from the shipped MXC audit example. - build the command, working directory, and filesystem grant from each run's ShareDir - pass the workload through --driver-config-json. - preserve the host CONNECT proxy configuration and conditional audit coverage for the future host_connect_proxy merge - require the workload output when determining the audit verdict. Signed-off-by: Akber Raza <akberr@nvidia.com> * fix(mxc): omit command arguments from OCSF audit logs - record only the executable basename for MXC CreateProcessInSandbox audit events - leave process.cmd_line unset so workload arguments cannot reach shorthand or JSONL logs - cover tokens, passwords, signed URLs, and PII with a secret-leak regression test - update the audit example, architecture guidance, and published logging documentation - preserve ETW attribution and future host_connect_proxy enforcement behavior Signed-off-by: Akber Raza <akberr@nvidia.com> * feat(etw): enhance ETW session management with distinct naming for concurrent gateways * fix(etw): bound the audit queue during overload - replace the unbounded ETW callback channel with count- and byte-bounded buffering - keep the ETW callback non-blocking and count records rejected during overload - emit immediate, rate-limited warnings that identify resulting audit coverage gaps - make the audit example fail when queue overload causes dropped ETW records - cover stalled consumers, oversized events, and warning throttling with unit tests Signed-off-by: Akber Raza <akberr@nvidia.com> * fix(etw): harden sandbox audit attribution - remove command-line and persistent per-PID fallback keys from live and replay resolution - retire the driver-owned wxc-exec PID before publishing child completion - retain established identity, activity, and correlation-vector links only for the five-second late-event window - prevent buffered records from crossing rapid PID retirement and reuse boundaries - add resolver and lifecycle coverage and document the attribution trust boundary Signed-off-by: Akber Raza <akberr@nvidia.com> * chore(mxc): address rebase follow-ups Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(mxc): align OCSF audit example with driver config - remove unsupported egress proxy settings - stop requiring the unavailable proxy audit event - update example documentation for supported event coverage Signed-off-by: Akber Raza <akberr@nvidia.com> * fix(etw): redact command-line secrets in DecodedEtwEvent summary * fix(etw): enhance PID resolution and event attribution logic for ETW records * fix(ocsf): restrict gateway-local JSONL sink to Windows/MXC path with opt-in configuration * address rebase issues * fix(mxc): address ETW audit review feedback Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(mxc): fail closed across ambiguous PID reuse Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(mxc): bind ETW attribution to process generation Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Akber Raza <akberr@nvidia.com> Signed-off-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: Jamie King <jamiek@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com>
253 lines
9.0 KiB
Plaintext
253 lines
9.0 KiB
Plaintext
---
|
|
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
title: "OCSF JSON Export"
|
|
sidebar-title: "OCSF JSON Export"
|
|
description: "How to enable full OCSF JSON logging for SIEM integration, compliance, and structured analysis. Includes schema version downgrade for AWS Security Lake, Splunk, and CrowdStrike compatibility."
|
|
keywords: "Generative AI, Cybersecurity, OCSF, JSON, SIEM, Compliance, Observability"
|
|
---
|
|
|
|
The [shorthand log format](/observability/logging) is optimized for humans and agents reading logs in real time. For machine consumption, compliance archival, or SIEM integration, you can enable full OCSF JSON export. This writes every OCSF event as a complete JSON record in JSONL format, one JSON object per line.
|
|
|
|
## Enable JSON Export
|
|
|
|
Use the `ocsf_json_enabled` setting to toggle JSON export. The setting can be applied globally, for all sandboxes, or per-sandbox.
|
|
|
|
Global:
|
|
|
|
```shell
|
|
openshell settings set --global --key ocsf_json_enabled --value true
|
|
```
|
|
|
|
Per-sandbox:
|
|
|
|
```shell
|
|
openshell settings set my-sandbox --key ocsf_json_enabled --value true
|
|
```
|
|
|
|
The setting takes effect on the next poll cycle, by default every 10 seconds. No sandbox restart is required.
|
|
|
|
To disable:
|
|
|
|
```shell
|
|
openshell settings set --global --key ocsf_json_enabled --value false
|
|
```
|
|
|
|
### Windows MXC gateways
|
|
|
|
MXC audit events originate in the Windows gateway rather than an in-sandbox
|
|
supervisor. Enable the MXC ETW consumer in `gateway.toml`, then explicitly opt
|
|
the gateway into durable JSONL output:
|
|
|
|
```toml
|
|
[openshell.drivers.mxc]
|
|
etw_audit = true
|
|
```
|
|
|
|
```powershell
|
|
$env:OPENSHELL_OCSF_JSON = "1"
|
|
# Optional; defaults to %PROGRAMDATA%\OpenShell\logs.
|
|
$env:OPENSHELL_OCSF_LOG_DIR = "D:\OpenShell\audit"
|
|
openshell-gateway --drivers mxc --config gateway.toml
|
|
```
|
|
|
|
The sink is available only when Windows selects the MXC driver. It rotates
|
|
daily and retains the three most recent files. Diagnostic `--log-level` values
|
|
do not suppress records in this explicitly enabled audit sink.
|
|
|
|
## Output Location
|
|
|
|
For sandbox supervisors, OCSF JSON records are written to
|
|
`/var/log/openshell-ocsf.YYYY-MM-DD.log` inside the sandbox. Windows MXC gateway
|
|
records use `%PROGRAMDATA%\OpenShell\logs` or `OPENSHELL_OCSF_LOG_DIR`. Both
|
|
sinks rotate daily and retain the three most recent files.
|
|
|
|
## JSON Record Structure
|
|
|
|
`metadata.uid` uniquely identifies an event and stays unchanged when that record is serialized again. Use `container.uid` to associate the event with its sandbox, not `metadata.uid`. Records without a sandbox association omit the container; unknown images are omitted rather than represented by an empty image name.
|
|
|
|
Each line is a complete OCSF v1.8.0 JSON object. Here is an example of a network connection event:
|
|
|
|
```json
|
|
{
|
|
"class_uid": 4001,
|
|
"class_name": "Network Activity",
|
|
"category_uid": 4,
|
|
"category_name": "Network Activity",
|
|
"activity_id": 1,
|
|
"activity_name": "Open",
|
|
"severity_id": 1,
|
|
"severity": "Informational",
|
|
"status_id": 1,
|
|
"status": "Success",
|
|
"time": 1775014138811,
|
|
"message": "CONNECT allowed api.github.com:443",
|
|
"metadata": {
|
|
"product": {
|
|
"name": "OpenShell Sandbox Supervisor",
|
|
"vendor_name": "NVIDIA",
|
|
"version": "0.3.0"
|
|
},
|
|
"version": "1.8.0"
|
|
},
|
|
"action_id": 1,
|
|
"action": "Allowed",
|
|
"disposition_id": 1,
|
|
"disposition": "Allowed",
|
|
"dst_endpoint": {
|
|
"domain": "api.github.com",
|
|
"port": 443
|
|
},
|
|
"src_endpoint": {
|
|
"ip": "10.42.0.31",
|
|
"port": 37494
|
|
},
|
|
"actor": {
|
|
"process": {
|
|
"name": "/usr/bin/curl",
|
|
"pid": 57
|
|
}
|
|
},
|
|
"firewall_rule": {
|
|
"name": "github_api",
|
|
"type": "opa"
|
|
}
|
|
}
|
|
```
|
|
|
|
And a denied connection:
|
|
|
|
```json
|
|
{
|
|
"class_uid": 4001,
|
|
"class_name": "Network Activity",
|
|
"activity_id": 1,
|
|
"activity_name": "Open",
|
|
"severity_id": 3,
|
|
"severity": "Medium",
|
|
"status_id": 2,
|
|
"status": "Failure",
|
|
"action_id": 2,
|
|
"action": "Denied",
|
|
"disposition_id": 2,
|
|
"disposition": "Blocked",
|
|
"status_detail": "no matching policy",
|
|
"message": "CONNECT denied httpbin.org:443",
|
|
"dst_endpoint": {
|
|
"domain": "httpbin.org",
|
|
"port": 443
|
|
},
|
|
"actor": {
|
|
"process": {
|
|
"name": "/usr/bin/curl",
|
|
"pid": 63
|
|
}
|
|
},
|
|
"firewall_rule": {
|
|
"name": "-",
|
|
"type": "opa"
|
|
}
|
|
}
|
|
```
|
|
|
|
<Note>
|
|
The JSON examples above are formatted for readability. The actual JSONL file contains one JSON object per line with no whitespace formatting.
|
|
</Note>
|
|
|
|
## OCSF Event Classes in JSON
|
|
|
|
The `class_uid` field identifies the event type:
|
|
|
|
| `class_uid` | Class | Shorthand prefix |
|
|
|---|---|---|
|
|
| 4001 | Network Activity | `NET:` |
|
|
| 4002 | HTTP Activity | `HTTP:` |
|
|
| 4007 | SSH Activity | `SSH:` |
|
|
| 1007 | Process Activity | `PROC:` |
|
|
| 2004 | Detection Finding | `FINDING:` |
|
|
| 5019 | Device Config State Change | `CONFIG:` |
|
|
| 6002 | Application Lifecycle | `LIFECYCLE:` |
|
|
|
|
## SIEM Schema Version Compatibility
|
|
|
|
OpenShell emits OCSF v1.8.0 events internally, but many SIEMs only support older schema versions. The `ocsf_schema_version` setting tells the JSONL layer to downgrade events before writing, stripping fields and profiles that don't exist in the target version.
|
|
|
|
Set the target version globally:
|
|
|
|
```shell
|
|
openshell settings set --global --key ocsf_schema_version --value "1.1"
|
|
```
|
|
|
|
Or per-sandbox:
|
|
|
|
```shell
|
|
openshell settings set my-sandbox --key ocsf_schema_version --value "1.3"
|
|
```
|
|
|
|
The setting takes effect on the next poll cycle, by default every 10 seconds. No sandbox restart is required.
|
|
|
|
Supported target versions are `1.1` and `1.3`. When set, the JSONL layer applies the following transformations:
|
|
|
|
- Strips fields added after the target version: `ai_model`, `container`, `observation_point_id`
|
|
- Removes unknown profile names from `metadata.profiles`: `ai_operation`, `container`
|
|
- Rewrites `metadata.version` to match the target version
|
|
|
|
The shorthand log output is unaffected. The internal event model stays at v1.8.0; only the serialized JSONL is transformed.
|
|
|
|
When unset or empty, no downgrade is applied and events are written at the current schema version.
|
|
|
|
| SIEM | Required OCSF Version | `ocsf_schema_version` Value |
|
|
|---|---|---|
|
|
| AWS Security Lake | v1.1.0 | `1.1` |
|
|
| Splunk CIM Add-On | v1.1-v1.3 | `1.1` or `1.3` |
|
|
| CrowdStrike FDR | v1.5.0 | Not yet supported |
|
|
| Datadog Cloud SIEM | v1.5.0 (selectable) | Not yet supported |
|
|
|
|
<Warning>
|
|
Downgrading is lossy. The `ai_model` field and `ai_operation` profile are stripped for v1.1 and v1.3 targets, so AI model attribution is not present in downgraded events. If your audit workflow requires knowing which model handled a request, use the full schema version (leave `ocsf_schema_version` unset). OpenShell supports OCSF v1.8.0 natively; upgrade your SIEM's OCSF schema if you need AI attribution alongside backward-compatible ingestion.
|
|
</Warning>
|
|
|
|
Downgraded events include a `downgraded_from` marker in the `unmapped` object so auditors can distinguish "no model was involved" from "model attribution was stripped":
|
|
|
|
```json
|
|
{
|
|
"unmapped": {
|
|
"downgraded_from": "1.8.0"
|
|
}
|
|
}
|
|
```
|
|
|
|
The core OCSF event structure (class UIDs, activity IDs, network and HTTP fields) is identical across v1.1 through v1.8. The stripped fields are all profile-gated additions.
|
|
|
|
## Integration with External Tools
|
|
|
|
The JSONL file can be shipped to any tool that accepts OCSF-formatted data:
|
|
|
|
| Tool | Integration Path |
|
|
|---|---|
|
|
| Splunk | Use the [Splunk OCSF Add-on](https://splunkbase.splunk.com/app/6943) to ingest OCSF JSONL files. Set `ocsf_schema_version` to `1.3` for CIM Add-On compatibility. |
|
|
| Amazon Security Lake | OCSF is the native schema for Security Lake. Set `ocsf_schema_version` to `1.1` for v1.1.0 compatibility. |
|
|
| CrowdStrike FDR | Ship JSONL files to Falcon Data Replicator. v1.5 downgrade target is not yet supported. |
|
|
| Elastic | Use Filebeat to ship JSONL files with the OCSF field mappings. |
|
|
| Custom pipelines | Parse the JSONL file with `jq`, Python, or any JSON-capable tool. |
|
|
|
|
Example with `jq` to extract all denied connections:
|
|
|
|
```shell
|
|
cat /var/log/openshell-ocsf.2026-04-01.log | \
|
|
jq -c 'select(.action == "Denied")'
|
|
```
|
|
|
|
## Relationship to Shorthand Logs
|
|
|
|
Sandbox events identify the sandbox environment with `device.type_id: 99` (Other) and `device.type: "Sandbox"`. The operating system is reported separately in `device.os.name`; the device type does not classify the underlying host or compute backend.
|
|
|
|
The shorthand format in `openshell.YYYY-MM-DD.log` and the JSON format in `openshell-ocsf.YYYY-MM-DD.log` are derived from the same OCSF events. The shorthand is a human-readable projection; the JSON is the full structured record when no schema downgrade is configured. When `ocsf_schema_version` is set, the JSON export is a lossy projection of the internal event model. Both formats are generated at the same time from the same event data.
|
|
|
|
The shorthand log is always active. The JSON export is opt-in through `ocsf_json_enabled`.
|
|
|
|
## Next Steps
|
|
|
|
- Learn how to [read the shorthand format](/observability/logging) for real-time monitoring.
|
|
- Refer to the [OCSF specification](https://schema.ocsf.io/) for the full schema reference.
|