Supervisor OCSF events reached only the sandbox's local JSONL file and
the shorthand log lines sent to the gateway, so collectors had no
structured copy of most of the security record. When the gateway has
`[openshell.gateway.otlp.ocsf]` configured, compute drivers now pass its
endpoint and schema version to supervisors, which export their own OCSF
events directly as OTLP log records. Supervisors already have egress,
so the gateway and its supervisor connection carry no extra load.
Supervisors use the same `openshell-otel` exporter as the gateway, with
a supervisor resource identity that includes `openshell.sandbox.id` and
`openshell.gateway.name`, so telemetry from sandboxes of different
gateways sharing a collector can be told apart. This applies to
supervisor spans too. Drivers pass the settings to supervisors as
`OPENSHELL_OTLP_OCSF_ENDPOINT`, `OPENSHELL_OTLP_OCSF_SCHEMA_VERSION`, and
`OPENSHELL_GATEWAY_NAME`; the gateway passes them to the VM driver as
arguments.
Collector headers that the gateway uses for OCSF export
(`OTEL_EXPORTER_OTLP_LOGS_HEADERS`, else `OTEL_EXPORTER_OTLP_HEADERS`)
travel in each sandbox's supervisor authentication bundle, which only
the supervisor reads, so authenticated collectors accept supervisor
records without exposing credentials to the workload.
The VM driver now sends SIGTERM and waits for the supervisor to exit
before closing its liveness pipe. Closing the pipe first made the
supervisor exit at once and drop queued records.
Closes#4544
Refs #2892
Signed-off-by: Kris Hicks <khicks@nvidia.com>
The gateway only persisted its OCSF events to the JSONL file, so collectors had
to tail that file. Adding `[openshell.gateway.otlp.ocsf]` now also exports each
gateway OCSF event as an OTLP log record, using the same resource identity as
gateway traces. `schema_version` optionally downgrades the exported JSON to
OCSF 1.1 or 1.3.
OTLP endpoints are now configured per signal, so OCSF events can go to a
collector that feeds a SIEM while spans go elsewhere. The existing
`[openshell.gateway.otlp]` `endpoint` is the default for every signal and on
its own still enables trace export. `[otlp.traces]` overrides the trace
endpoint, and `[otlp.ocsf]` takes its own `endpoint` or falls back to the
default.
Each record carries the OCSF JSON as its body, the event time as its timestamp,
a severity mapped from `severity_id`, and `ocsf.*` and `openshell.sandbox.id`
attributes for filtering without parsing the body.
Records flow through a queue bounded by count and bytes to the SDK OTLP log
exporter in batches bounded the same way. Failed exports retry with backoff;
records lost to a full queue, an oversized body, or exhausted retries are
counted in metrics and reported to the collector in an `openshell.ocsf.gap`
record on the next successful export. The exporter lives in `openshell-otel` so
other processes can share it.
Helm deployments set the `traces` and `ocsf` tables through `gatewayConfig`,
and the local development gateway tasks enable OCSF export.
Closes#4499
Refs #2892
Signed-off-by: Kris Hicks <khicks@nvidia.com>
When the gateway exports OTLP traces, compute drivers pass the gateway's
endpoint to supervisors as OPENSHELL_OTLP_ENDPOINT, along with TRACEPARENT
from the operation that launched them. The Kubernetes, Docker, Podman, and
VM drivers all receive the endpoint from the gateway; driver TOML cannot
set it. On Podman Machine, the Podman driver points a loopback endpoint
at host.containers.internal, since supervisor loopback is the VM's.
The supervisor exports spans as openshell-supervisor, tagged with
the sandbox ID, and flushes them before exiting.
supervisor.startup joins the sandbox's creation trace and covers image
policy discovery, policy load, and boundary attach, confirm, agent start,
and access start. Supervisor calls to the gateway carry W3C trace context,
so the gateway's server spans nest under them.
Each egress connection emits supervisor.egress.connect with authorize,
resolve, and dial children. These spans use DEBUG level because every
outbound connection starts its own trace. OpenShell spans export at INFO,
or at the sandbox log level when it is debug or trace; spans from other
libraries export at INFO.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Centralize compute-driver RPC descriptors, stream instrumentation, provider
routing, and standalone installation in openshell-otel. Use typed RPC
constants so gateway and in-process driver paths cannot panic on unknown
operation strings or repeat runtime method parsing.
Emit semantic-convention rpc.service and rpc.method attributes, preserve
trace context and resource identity across deployment modes, and route both
RPC boundary and backend crate spans to each selected driver provider. Leave
consumer-dropped watch spans unset while recording observed terminal status,
and avoid reboxing untraced external-driver streams.
Derive each driver tracing identity from Cargo package and crate metadata and
attach its descriptor to the compute-driver registration, keeping provider
selection and target routing tied to the registered implementation. Share
tracing setup and round-trip test support across Docker, Podman, Kubernetes,
and VM, and update the gateway tracing documentation.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Mirror the VM, Podman, and Docker driver tracing setup for Kubernetes.
Export standalone driver spans through OTLP/gRPC as the distinct
openshell-driver-kubernetes service, preserve gateway trace context, record
lifecycle operations and gRPC failures, and flush spans on shutdown.
Kubernetes currently runs in-process when selected as a built-in gateway
driver. Use the temporary server-boundary shim shared with Podman and Docker
so traces retain the shape they will have when Kubernetes moves to a
separate process. Move the common ComputeDriver RPC tracing layer into
openshell-otel to keep all drivers aligned.
Propagate the active W3C context through the controller-reserved Sandbox
annotation and enable Agent Sandbox OTLP export in the local k3s workflow.
This connects asynchronous controller reconciliation spans to the originating
OpenShell create trace.
Expose gateway OTLP configuration through Helm and add an Aspire collector
to the local k3s workflow. Extend helm:k3s:forward with OTLP ingest and trace
UI forwarding for Kubernetes and local container gateway development.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Mirror the VM and Podman driver tracing setup for Docker. Export Docker
driver spans through OTLP/gRPC as the distinct openshell-driver-docker
service, preserve gateway trace context, record lifecycle and asynchronous
provisioning operations, and report gRPC failures.
Docker currently runs in-process when selected as a built-in gateway driver.
Add the same temporary server-boundary shim used by Podman so traces retain
the shape they will have when Docker moves to a separate process. Generalize
the gateway provider selection for both in-process drivers and share the OTLP
collector fixture across Docker, Podman, and VM tracing tests.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Mirror the VM driver tracing setup for Podman. Export standalone driver
spans through OTLP/gRPC as the distinct openshell-driver-podman service,
propagate W3C context across ComputeDriver RPCs, record bounded RPC names
and failures, and flush buffered spans during graceful shutdown.
Podman still runs in-process when selected as a built-in gateway driver.
Add a temporary tracing shim that partitions gateway and Podman spans by
target into separate tracer providers while preserving their shared trace
and parentage. The shim also emits the same ComputeDriver server boundary
that the tonic layer emits out of process, keeping the observable trace
shape stable when Podman is eventually extracted.
Trace container create preparation, image and storage setup, lifecycle
operations, and cleanup. Document the service boundary and cover it with
isolated and repeated tracing tests.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Continue distributed traces across the gateway-to-driver process boundary and
export VM driver spans to the same OTLP/gRPC collector. The driver reports as
the distinct openshell-driver-vm service.
Updated the gateway architecture and configuration reference with a generic
external-driver forwarding contract.
Instrumented:
- Every RemoteComputeDriver RPC injects the active W3C trace context into
tonic metadata. Managed VM readiness and runtime initialization give startup
capability probes stable parent operations rather than isolated root spans.
- A tonic service layer creates fixed, low-cardinality server spans for every
ComputeDriver RPC. New handlers inherit tracing automatically; failures record
OpenTelemetry error status and the gRPC status code.
- Background provisioning remains attached to CreateSandbox after the RPC
returns without extending the RPC span lifetime.
- Provisioning records image preparation, bootstrap image resolution, overlay
preparation, lifecycle configuration, pre-launch hooks, guest preparation,
and launcher spawn as child spans.
- VM startup reconciliation roots one trace for the persisted-sandbox scan,
with per-sandbox restore and provision operations beneath it. The root remains
open until all spawned restore tasks finish.
- Delete cleanup records its own child operation.
Design notes:
- The gateway forwards its configured OTLP endpoint to managed external
drivers. SDK `OTEL_*` variables continue to own sampling, batching, limits,
headers, and transport tuning.
- The VM driver has its own tracer provider and service resource so trace
backends preserve the service boundary.
- RPC operation names come from an explicit method mapping, keeping cardinality
bounded without parsing the protobuf descriptor set at runtime.
- Propagation uses a remote SpanContext for spawned provisioning. This keeps
one trace while allowing the CreateSandbox server span to finish when the
RPC response is sent.
- Startup restoration is independent of gateway requests. It begins at the VM
driver reconciliation span rather than attaching to an unrelated RPC.
- Existing tracing events remain on the logging path. The OpenTelemetry layer
exports spans only and excludes the SDK exporter callsites to avoid recursive
traces.
- Export configuration failures do not prevent the driver from serving, and
buffered spans are drained during graceful shutdown.
- Trace fields identify drivers, sandboxes, images, lifecycle phases, and gRPC
outcomes without recording credentials, sandbox tokens, or request query
parameters.
Refs #2507
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Extract common OpenTelemetry provider construction into openshell-otel so
OpenShell services share one OTLP/gRPC export implementation.
The shared crate owns:
- typed exporter setup errors and non-fatal provider enablement;
- endpoint trimming and URI validation before lazy exporter connection;
- fixed and environment-or-default service-name policies;
- service version and caller-supplied resource attributes;
- batch tracer-provider construction; and
- span-only tracing layers that exclude OpenTelemetry exporter callsites.
Migrate the gateway to the shared provider while retaining its configurable
service name, error marking, and tracing test collector. Add the shared crate
to the architecture inventory and document the tracing boundary.
Refs #2507
Signed-off-by: Kris Hicks <khicks@nvidia.com>
krishicks