Files
OpenShell/crates/openshell-sandbox
Roland Huss fdce6ef260 feat(supervisor): relay agent OTLP traces to the collector (#4276)
* feat(supervisor): relay agent OTLP traces to the collector

Serve the reserved endpoint 192.0.0.8:4318 inside the supervisor, attribute each batch with openshell.sandbox.id and openshell.telemetry.source=agent at the protobuf wire level, buffer it with fixed bounds, and forward it over OTLP/gRPC to the collector the supervisor already uses, flushing before the exit report and keeping the relay up for exec sessions.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* refactor(supervisor): let the OTLP relay exporter take a prepared channel

Expose Exporter::with_channel as the seam for a collector transport built once in openshell-otel and shared by the gateway, the supervisor's own spans, and the relay. Exporter::new keeps building the plaintext channel for now.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* feat(sandbox): leave caller-set OTLP endpoints alone

Inject the relay variables only when the sandbox creation request names no OTLP endpoint, generic or traces-specific. A caller who names one has chosen where the agent exports; overwriting it would silently redirect an agent that deliberately exports elsewhere. A protocol or compression setting alone does not suppress injection. Applies to both spawn paths and to exec children, with the docs updated to match.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* fix(supervisor): address OTLP relay review feedback

Inject only the generic OTEL_EXPORTER_OTLP_ENDPOINT and
OTEL_EXPORTER_OTLP_PROTOCOL as lowest-precedence defaults, and only
while the relay is active. The supervisor now starts the relay before
attach and signals it to the sandbox runtime through
AgentSpec::otlp_relay_active. An endpoint in the creation request, the
image ENV, or an exec environment suppresses the defaults, and a
provider-supplied traces endpoint is removed.

Harden the relay itself:
- skip the exit flush and shutdown wait when the last export failed
- count batches still pending on Drop as discarded and log them
- validate the collector endpoint with the shared openshell-otel helper
- accept Content-Encoding: identity
- fix the 413 and shutdown 503 messages and the memory bound docs
- narrow the attribution claim to resource attributes

Document best-effort delivery and the 404 for metrics and logs.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* feat(supervisor): gate the agent trace relay behind [otlp.agent]

The relay started whenever a sandbox supervisor had a trace endpoint, so
enabling gateway trace export also changed every workload's environment.
Make it opt-in: `[openshell.gateway.otlp.agent]` with `enabled = true`
turns it on, and its optional `endpoint` falls back to the shared
`[openshell.gateway.otlp]` endpoint, like the OCSF table.

The gateway validates the endpoint and drivers pass it to supervisors as
OPENSHELL_OTLP_AGENT_ENDPOINT, separate from OPENSHELL_OTLP_ENDPOINT.
Docker, Podman (including the Podman Machine host alias), Kubernetes and
VM drivers carry it, in-process and as standalone binaries. The shared
Kubernetes supervisor still does not relay.

The relay now acknowledges OTLP metrics and logs exports with an empty
success and discards them, so agent SDKs that export every signal to the
generic endpoint stop retrying. Discards are counted and logged once.

Local dev gateway tasks, the Skaffold values and the e2e harness enable
the table; OPENSHELL_E2E_OTLP_AGENT=0 runs the inert case with trace
export still on.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* docs(supervisor): describe the agent trace relay architecture

Add an "Agent traces" section to the architecture page and document the
relay in the crate READMEs: the reserved-destination hook in the network
supervisor, the OTEL defaults the sandbox runtime injects, and how the
Docker, Podman, Kubernetes, and VM drivers pass the agent endpoint.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* fix(sandbox): disable OTLP compression in the relay env and cover exec children

The relay refuses compressed bodies, so an image or creation request that
sets gzip compression without naming an endpoint received the relay default
and lost every trace export to a 415. The relay default now also sets
OTEL_EXPORTER_OTLP_COMPRESSION and OTEL_EXPORTER_OTLP_TRACES_COMPRESSION to
none, as #4275 asks.

Boundary exec children start from a cleared environment and never see the
image ENV, yet the relay default was suppressed when the sandbox's own
environment named an endpoint, which left exec children with no endpoint at
all. apply_otlp_relay_env now takes a ChildEnvironment so only children that
inherit the sandbox environment consult it.

Add a probe test for both TLS environment modes together with the relay
default, now that the mode-aware TLS layering lives in
compose_canonical_process_environment, and describe the whole composition
at its call sites.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* fix(supervisor): flush the agent trace relay before suspending

The suspend lifecycle from #4765 returned without shutting the relay down,
so Drop discarded batches the relay had already acknowledged to the agent,
which does not resend them after resume. Shut the relay down, bounded by
FLUSH_TIMEOUT, once the workload is frozen and before networking is
dropped.

Also tidy what the rebases left behind: drop a stray blank line inside the
upstream startup block, rename SandboxInstanceConfig::agent_otlp_endpoint
to otlp_agent_endpoint to match every other identifier, serialize the OCSF
table through the generic serialize_enabled, correct the UnsupportedScheme
doc comment (the supervisor's own exporter does speak TLS), and build the
reserved-destination proxy test identities with host-native paths.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* test(e2e): scope the Podman OTLP default and restore upstream lockfiles

The Podman wrapper enabled OTLP export and the agent relay for every e2e
run, including single-target CI jobs such as additional_ca. Enable it only
for the full suite or a selection that includes otlp_relay, following the
SPIFFE fixture pattern; the Docker wrapper stays opt-in.

The e2e and features lockfiles had downgraded libsqlite3-sys,
crossbeam-queue and hybrid-array against upstream with no manifest change
requiring it. Restore upstream's lockfiles and add only the packages the
new e2e dependencies pull in.

Signed-off-by: Roland Huß <rhuss@redhat.com>

---------

Signed-off-by: Roland Huß <rhuss@redhat.com>
2026-10-28 15:26:52 +00:00
..

Sandbox runtime

openshell-sandbox owns the Linux workload isolation mechanisms and their kernel qualification probes. The linux module contains seccomp notifications and child filters, task-memory access, process-signal mediation, procfs descriptor lookup, socket identity, Landlock qualification, and the workload launch thread.

sandbox::linux composes policy enforcement. The top-level linux module owns the lower-level mechanisms used by that orchestration and the boundary server. Unsafe kernel ABI, process-launch, and descriptor-ownership operations stay in this runtime crate behind safe APIs where their contracts can be enforced.

The network broker retains source socket descriptors only while preparing a workload endpoint. A successful connection or listen releases that reference; the kernel then tracks the lifetime through workload-owned descriptors. Native accept and repeated listen preserve the socket's loopback confinement. Closing the last workload reference to a listener removes it without waiting for registry collection. Broker source descriptors always use close-on-exec independently of the workload's requested descriptor flags.

A boundary configured with RuntimeInitiatedTlsTcp binds no listener. It dials only the driver-provided supervisor addresses, which must be concrete, non-loopback, and share one port, and serves TLS and gRPC over that outbound connection. It keeps one spare connection that the supervisor has not yet used and dials a replacement when the supervisor starts HTTP/2 on it or the spare closes. Live connections keep the same disconnect, freeze, and recovery-deadline handling as listener mode, and the workload still cannot connect to the control port.

A runtime of a supervisor shared by several sandboxes (one with a route handle) names its route in a version 2 preface on every dial and follows the supervisors' redirects, on the same port and never on loopback, at most two hops. A supervisor that closes the negotiation without replying, as one from before version 2 does, is dialed again with a version 1 preface and treated as a lone supervisor; this changes only routing, since TLS pinning, request authentication, and takeover grants are unaffected. It keeps a spare at the member whose connection it last confirmed, its authenticated owner; an unauthenticated accept never makes a member the owner. Once a supervisor names itself a pool member or redirects it, the runtime also dials the configured addresses past every known member every 15 seconds, up to three times, so a member that hosts the route without a spare gets one: a previous owner that stays reachable after the gateway granted the route elsewhere, holding an unused spare, cannot keep the new owner from the runtime. A lone shared supervisor never names itself, so it holds the one spare and is never probed. The runtime keeps at most three unused spares, one per member, and never closes an established connection to refresh routing. A member's address grants nothing: a later term at a fenced owner's address authenticates with its own grant. A retired connection that does not shut down gracefully within two seconds, such as an unused spare, has its transport dropped.

When AgentSpec::otlp_relay_active is set, child_env::apply_otlp_relay_env gives every workload process (main, exec, and SSH) OTEL_EXPORTER_OTLP_ENDPOINT pointing at the relay address, OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf, and none for OTEL_EXPORTER_OTLP_COMPRESSION and OTEL_EXPORTER_OTLP_TRACES_COMPRESSION, and removes any inherited OTEL_EXPORTER_OTLP_TRACES_ENDPOINT. These defaults apply after the provider variables. If the creation request or a per-exec environment names a generic or traces-specific OTLP endpoint, the runtime injects nothing, and the workload exports to that endpoint under the sandbox's network policy. An endpoint in the image ENV does the same for the main process; exec children start from a cleared environment (ChildEnvironment::Cleared), never see the image's endpoint, and still get the relay default.

Supervisor takeover

The boundary advertises openshell.sandbox.supervisor-takeover. A replacement supervisor process may attach to a running generation only with a gateway-signed takeover grant for a later credential epoch, its bearer of that epoch, and the grant's supervisor process identity. The attach is staged and promoted at confirmation, which advances the accepted epoch, pins the new process, and closes every connection of the previous owner; credentials of earlier epochs are refused from then on. Refused attempts emit an OCSF configuration-state event and a sandbox-supervisor-takeover-denied finding.

Attach and confirmation are serialized, and the registry checks an attach before it binds anything. Owner operations (exec, signals, termination, provider updates, network decisions, DNS answers) hold an ownership guard from authorization through their effect, and a promotion takes it exclusively, so none of the replaced owner's operations takes effect afterwards. The guard is asynchronous: tasks on the runtime's workers wait for it without blocking them. A confirmation replay on the connection in force promotes nothing and skips the guard. Promotion and the supervisory state change together, so a replacement lost mid-promotion leaves the workload frozen under a fresh recovery deadline. Each supervisor offers a recovery deadline when it attaches (10 to 300 seconds; the runtime refuses any other before binding anything), and the deadline of the connection in force when supervision is lost bounds the freeze: 30 seconds without an offer. A supervisor pool offers 60 seconds unless the gateway configures another.

The runtime keeps the trust-on-first-use executable pins of the generation (binary_pins): the first digest it offers for each path, from its own measurements. A contradicting identity is offered as a resolution failure, which every supervisor denies, so a replacement with an empty cache cannot pin a binary the workload replaced.