mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-16 05:20:22 +08:00
John T. Myers
b7779bdefa
* feat(sandbox): integrate OCSF structured logging for all sandbox events WIP: Replace ad-hoc tracing calls with OCSF event builders across all sandbox subsystems (network, SSH, process, filesystem, config, lifecycle). - Register ocsf_logging_enabled setting (defaults false) - Replace stdout/file fmt layers with OcsfShorthandLayer - Add conditional OcsfJsonlLayer for /var/log/openshell-ocsf.log - Update LogPushLayer to extract OCSF shorthand for gRPC push - Migrate ~106 log sites to OCSF builders (NetworkActivity, HttpActivity, SshActivity, ProcessActivity, DetectionFinding, ConfigStateChange, AppLifecycle) - Add openshell-ocsf to all Docker build contexts * fix(scripts): attach provider to all smoke test phases to avoid rate limits GitHub's unauthenticated API rate limit (60/hour) causes flaky 403s for Phases 1, 2, and 4. Fix by attaching the provider to all sandboxes and upgrading the Phase 1 policy to L7 so credential injection works. Phase 4 (tls:skip) cannot inject credentials by design, so relax the assertion to accept either 200 or 403 from upstream -- both prove the proxy forwarded the request. * fix(ocsf): remove timestamp from shorthand format to avoid double-timestamp The display layer (gateway logs, TUI, sandbox logs CLI) already prepends a timestamp. Having one in the shorthand output too produces redundant double-timestamps like: 15:49:11 sandbox INFO 15:49:11.649 I NET:OPEN ALLOWED ... Now the shorthand is just the severity + structured content: 15:49:11 sandbox INFO I NET:OPEN ALLOWED ... * refactor(ocsf): replace single-char severity with bracketed labels Replace cryptic single-character severity codes (I/L/M/H/C/F) with readable bracketed labels: [LOW], [MED], [HIGH], [CRIT], [FATAL]. Informational severity (the happy-path default) is omitted entirely to keep normal log output clean and avoid redundancy with the tracing-level INFO that the display layer already provides. Before: sandbox INFO I NET:OPEN ALLOWED ... After: sandbox INFO NET:OPEN ALLOWED ... Before: sandbox INFO M NET:OPEN DENIED ... After: sandbox INFO [MED] NET:OPEN DENIED ... * feat(sandbox): use OCSF level label for structured events in log push Set the level field to 'OCSF' instead of 'INFO' for OCSF events in the gRPC log push. This visually distinguishes structured OCSF events from plain tracing output in the TUI and CLI sandbox logs: sandbox OCSF NET:OPEN [INFO] ALLOWED python3(42) -> api.example.com:443 sandbox OCSF NET:OPEN [MED] DENIED python3(42) -> blocked.com:443 sandbox INFO Fetching sandbox policy via gRPC * fix(sandbox): convert new Landlock path-skip warning to OCSF PR #677 added a warn!() for inaccessible Landlock paths in best-effort mode. Convert to ConfigStateChangeBuilder with degraded state so it flows through the OCSF shorthand format consistently. * fix(sandbox): use rolling appender for OCSF JSONL file Match the main openshell.log rotation mechanics (daily, 3 files max) instead of a single unbounded append-only file. Prevents disk exhaustion when ocsf_logging_enabled is left on in long-running sandboxes. * fix(sandbox): address reviewer warnings for OCSF integration W1: Remove redundant 'OCSF' prefix from shorthand file layer — the class name (NET:OPEN, HTTP:GET) already identifies structured events and the LogPushLayer separately sets the level field. W2: Log a debug message when OCSF_CTX.set() is called a second time instead of silently discarding via let _. W3: Document the boundary between OCSF-migrated events and intentionally plain tracing calls (DEBUG/TRACE, transient, internal plumbing). W4: Migrate remaining iptables LOG rule failure warnings in netns.rs (IPv4 TCP/UDP, IPv6 TCP/UDP) to ConfigStateChangeBuilder for consistency with the IPv4 bypass rule failure already migrated. W5: Migrate malformed inference request warn to NetworkActivity with ActivityId::Refuse and SeverityId::Medium. W6: Use Medium severity for L7 deny decisions (both CONNECT tunnel and FORWARD proxy paths) to match the CONNECT deny severity pattern. Allows and audits remain Informational. * refactor(sandbox): rename ocsf_logging_enabled to ocsf_json_enabled The shorthand logs are already OCSF-structured events. The setting specifically controls the JSONL file export, so the name should reflect that: ocsf_json_enabled. * fix(ocsf): add timestamps to shorthand file layer output The OcsfShorthandLayer writes directly to the log file with no outer display layer to supply timestamps. Add a UTC timestamp prefix to every line so the file output matches what tracing::fmt used to provide. Before: CONFIG:VALIDATED [INFO] Validated 'sandbox' user exists in image After: 2026-04-01T15:49:11.649Z CONFIG:VALIDATED [INFO] Validated ... * fix(docker): touch openshell-ocsf source to invalidate cargo cache The supervisor-workspace stage touches sandbox and core sources to force recompilation over the rust-deps dummy stubs, but openshell-ocsf was missing. This caused the Docker cargo cache to use stale ocsf objects from the deps stage, preventing changes to the ocsf crate (like the timestamp fix) from appearing in the final binary. Also adds a shorthand layer test verifying timestamp output, and drafts the observability docs section. * fix(ocsf): add OCSF level prefix to file layer shorthand output Without a level prefix, OCSF events in the log file have no visual anchor at the position where standard tracing lines show INFO/WARN. This makes scanning the file harder since the eye has nothing consistent to lock onto after the timestamp. Before: 2026-04-01T04:04:13.065Z CONFIG:DISCOVERY [INFO] ... After: 2026-04-01T04:04:13.065Z OCSF CONFIG:DISCOVERY [INFO] ... * fix(ocsf): clean up shorthand formatting for listen and SSH events - Fix double space in NET:LISTEN, SSH:LISTEN, and other events where action is empty (e.g., 'NET:LISTEN [INFO] 10.200.0.1' -> 'NET:LISTEN [INFO] 10.200.0.1') - Add listen address to SSH:LISTEN event (was empty) - Downgrade SSH handshake intermediate steps (reading preface, verifying) from OCSF events to debug!() traces. Only the final verdict (accepted/denied) is an OCSF event now, reducing noise from 3 events to 1 per SSH connection. - Apply same spacing fix to HTTP shorthand for consistency. * docs(observability): update examples with OCSF prefix and formatting fixes Align doc examples with the deployed output: - Add OCSF level prefix to all shorthand examples in the log file - Show mixed OCSF + standard tracing in the file format section - Update listen events (no double space, SSH includes address) - Show one SSH:OPEN per connection instead of three - Update grep patterns to use 'OCSF NET:' etc. * docs(agents): add OCSF logging guidance to AGENTS.md Add a Sandbox Logging (OCSF) section to AGENTS.md so agents have in-context guidance for deciding whether new log emissions should use OCSF structured logging or plain tracing. Covers event class selection, severity guidelines, builder API usage, dual-emit pattern for security findings, and the no-secrets rule. Also adds openshell-ocsf to the Architecture Overview table. * fix: remove workflow files accidentally included during rebase These files were already merged to main in separate PRs. They got pulled into our branch during rebase conflict resolution for the deleted docs-preview-pr.yml file. * docs(observability): use sandbox connect instead of raw SSH Users access sandboxes via 'openshell sandbox connect', not direct SSH. * fix(docs): correct settings CLI syntax in OCSF JSON export page The settings CLI requires --key and --value named flags, not positional arguments. Also fix the per-sandbox form: the sandbox name is a positional argument, not a --sandbox flag. * fix(e2e): update log assertions for OCSF shorthand format The E2E tests asserted on the old tracing::fmt key=value format (action=allow, l7_decision=audit, FORWARD, L7_REQUEST, always-blocked). Update to match the new OCSF shorthand (ALLOWED/DENIED, HTTP:, NET:, engine:ssrf, policy:). * feat(sandbox): convert WebSocket upgrade log calls to OCSF PR #718 added two log calls for WebSocket upgrade handling: - 101 Switching Protocols info → NetworkActivity with Upgrade activity. This is a significant state change (L7 enforcement drops to raw relay). - Unsolicited 101 without client Upgrade header → DetectionFinding with High severity. A non-compliant upstream sending 101 without a client Upgrade request could be attempting to bypass L7 inspection.
110 lines
5.0 KiB
Docker
110 lines
5.0 KiB
Docker
# syntax=docker/dockerfile:1.6
|
|
|
|
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
|
|
|
|
ARG OSXCROSS_IMAGE=crazymax/osxcross:latest
|
|
ARG PYTHON_VERSION=3.12
|
|
|
|
FROM ${OSXCROSS_IMAGE} AS osxcross
|
|
|
|
FROM python:${PYTHON_VERSION}-slim AS builder
|
|
|
|
ARG TARGETARCH
|
|
ARG CARGO_TARGET_CACHE_SCOPE=default
|
|
|
|
ENV PATH="/root/.cargo/bin:/usr/local/bin:/osxcross/bin:${PATH}"
|
|
ENV LD_LIBRARY_PATH="/osxcross/lib"
|
|
|
|
COPY --from=osxcross /osxcross /osxcross
|
|
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
build-essential \
|
|
ca-certificates \
|
|
clang \
|
|
curl \
|
|
libssl-dev \
|
|
pkg-config \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# aws-lc-sys probes with --target=arm64-apple-macosx and clang then looks for
|
|
# arm64-apple-macosx-ld. Provide a linker alias to osxcross ld64.
|
|
RUN ln -sf /osxcross/bin/arm64-apple-darwin25.1-ld /usr/local/bin/arm64-apple-macosx-ld
|
|
|
|
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
|
|
RUN rustup target add aarch64-apple-darwin
|
|
RUN pip install --no-cache-dir maturin
|
|
|
|
WORKDIR /build
|
|
|
|
ENV CC_aarch64_apple_darwin=oa64-clang
|
|
ENV CXX_aarch64_apple_darwin=oa64-clang++
|
|
ENV AR_aarch64_apple_darwin=aarch64-apple-darwin25.1-ar
|
|
ENV CARGO_TARGET_AARCH64_APPLE_DARWIN_LINKER=oa64-clang
|
|
ENV CARGO_TARGET_AARCH64_APPLE_DARWIN_AR=aarch64-apple-darwin25.1-ar
|
|
|
|
# Copy dependency manifests first for better caching.
|
|
COPY Cargo.toml Cargo.lock ./
|
|
COPY crates/openshell-cli/Cargo.toml crates/openshell-cli/Cargo.toml
|
|
COPY crates/openshell-core/Cargo.toml crates/openshell-core/Cargo.toml
|
|
COPY crates/openshell-ocsf/Cargo.toml crates/openshell-ocsf/Cargo.toml
|
|
COPY crates/openshell-providers/Cargo.toml crates/openshell-providers/Cargo.toml
|
|
COPY crates/openshell-router/Cargo.toml crates/openshell-router/Cargo.toml
|
|
COPY crates/openshell-sandbox/Cargo.toml crates/openshell-sandbox/Cargo.toml
|
|
COPY crates/openshell-server/Cargo.toml crates/openshell-server/Cargo.toml
|
|
COPY crates/openshell-bootstrap/Cargo.toml crates/openshell-bootstrap/Cargo.toml
|
|
COPY crates/openshell-core/build.rs crates/openshell-core/build.rs
|
|
COPY proto/ proto/
|
|
|
|
# Create dummy source files to build dependencies.
|
|
RUN mkdir -p crates/openshell-cli/src crates/openshell-core/src crates/openshell-ocsf/src crates/openshell-providers/src crates/openshell-router/src crates/openshell-sandbox/src crates/openshell-server/src crates/openshell-bootstrap/src && \
|
|
echo "fn main() {}" > crates/openshell-cli/src/main.rs && \
|
|
echo "fn main() {}" > crates/openshell-sandbox/src/main.rs && \
|
|
echo "fn main() {}" > crates/openshell-server/src/main.rs && \
|
|
touch crates/openshell-core/src/lib.rs && \
|
|
touch crates/openshell-ocsf/src/lib.rs && \
|
|
touch crates/openshell-providers/src/lib.rs && \
|
|
touch crates/openshell-router/src/lib.rs && \
|
|
touch crates/openshell-bootstrap/src/lib.rs
|
|
|
|
# Build dependencies only (cached unless Cargo.toml/lock changes).
|
|
RUN --mount=type=cache,id=cargo-registry-python-wheels-macos-${TARGETARCH},sharing=locked,target=/root/.cargo/registry \
|
|
--mount=type=cache,id=cargo-git-python-wheels-macos-${TARGETARCH},sharing=locked,target=/root/.cargo/git \
|
|
--mount=type=cache,id=cargo-target-python-wheels-macos-${TARGETARCH}-${CARGO_TARGET_CACHE_SCOPE},sharing=locked,target=/build/target \
|
|
cargo build --release --target aarch64-apple-darwin -p openshell-cli 2>/dev/null || true
|
|
|
|
# Copy actual source code and Python packaging files.
|
|
COPY crates/ crates/
|
|
COPY pyproject.toml README.md ./
|
|
COPY python/ python/
|
|
|
|
# Touch source files to ensure they're rebuilt (not the cached dummy).
|
|
# Touch build.rs and proto files to force proto code regeneration when the
|
|
# cargo target cache mount retains stale OUT_DIR artifacts from prior builds.
|
|
RUN touch crates/openshell-cli/src/main.rs \
|
|
crates/openshell-cli/src/lib.rs \
|
|
crates/openshell-bootstrap/src/lib.rs \
|
|
crates/openshell-core/src/lib.rs \
|
|
crates/openshell-providers/src/lib.rs \
|
|
crates/openshell-router/src/lib.rs \
|
|
crates/openshell-sandbox/src/main.rs \
|
|
crates/openshell-server/src/main.rs \
|
|
crates/openshell-core/build.rs \
|
|
proto/*.proto
|
|
|
|
# Declare version ARGs here (not earlier) so the git-hash-bearing values do not
|
|
# invalidate the expensive dependency-build layers above on every commit.
|
|
ARG OPENSHELL_CARGO_VERSION
|
|
ARG OPENSHELL_IMAGE_TAG
|
|
RUN --mount=type=cache,id=cargo-registry-python-wheels-macos-${TARGETARCH},sharing=locked,target=/root/.cargo/registry \
|
|
--mount=type=cache,id=cargo-git-python-wheels-macos-${TARGETARCH},sharing=locked,target=/root/.cargo/git \
|
|
--mount=type=cache,id=cargo-target-python-wheels-macos-${TARGETARCH}-${CARGO_TARGET_CACHE_SCOPE},sharing=locked,target=/build/target \
|
|
if [ -n "${OPENSHELL_CARGO_VERSION:-}" ]; then \
|
|
sed -i -E '/^\[workspace\.package\]/,/^\[/{s/^version[[:space:]]*=[[:space:]]*".*"/version = "'"${OPENSHELL_CARGO_VERSION}"'"/}' Cargo.toml; \
|
|
fi && \
|
|
maturin build --release --target aarch64-apple-darwin --out /wheels
|
|
|
|
FROM scratch AS wheels
|
|
COPY --from=builder /wheels/*.whl /
|