* feat(supervisor): relay agent OTLP traces to the collector Serve the reserved endpoint 192.0.0.8:4318 inside the supervisor, attribute each batch with openshell.sandbox.id and openshell.telemetry.source=agent at the protobuf wire level, buffer it with fixed bounds, and forward it over OTLP/gRPC to the collector the supervisor already uses, flushing before the exit report and keeping the relay up for exec sessions. Signed-off-by: Roland Huß <rhuss@redhat.com> * refactor(supervisor): let the OTLP relay exporter take a prepared channel Expose Exporter::with_channel as the seam for a collector transport built once in openshell-otel and shared by the gateway, the supervisor's own spans, and the relay. Exporter::new keeps building the plaintext channel for now. Signed-off-by: Roland Huß <rhuss@redhat.com> * feat(sandbox): leave caller-set OTLP endpoints alone Inject the relay variables only when the sandbox creation request names no OTLP endpoint, generic or traces-specific. A caller who names one has chosen where the agent exports; overwriting it would silently redirect an agent that deliberately exports elsewhere. A protocol or compression setting alone does not suppress injection. Applies to both spawn paths and to exec children, with the docs updated to match. Signed-off-by: Roland Huß <rhuss@redhat.com> * fix(supervisor): address OTLP relay review feedback Inject only the generic OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL as lowest-precedence defaults, and only while the relay is active. The supervisor now starts the relay before attach and signals it to the sandbox runtime through AgentSpec::otlp_relay_active. An endpoint in the creation request, the image ENV, or an exec environment suppresses the defaults, and a provider-supplied traces endpoint is removed. Harden the relay itself: - skip the exit flush and shutdown wait when the last export failed - count batches still pending on Drop as discarded and log them - validate the collector endpoint with the shared openshell-otel helper - accept Content-Encoding: identity - fix the 413 and shutdown 503 messages and the memory bound docs - narrow the attribution claim to resource attributes Document best-effort delivery and the 404 for metrics and logs. Signed-off-by: Roland Huß <rhuss@redhat.com> * feat(supervisor): gate the agent trace relay behind [otlp.agent] The relay started whenever a sandbox supervisor had a trace endpoint, so enabling gateway trace export also changed every workload's environment. Make it opt-in: `[openshell.gateway.otlp.agent]` with `enabled = true` turns it on, and its optional `endpoint` falls back to the shared `[openshell.gateway.otlp]` endpoint, like the OCSF table. The gateway validates the endpoint and drivers pass it to supervisors as OPENSHELL_OTLP_AGENT_ENDPOINT, separate from OPENSHELL_OTLP_ENDPOINT. Docker, Podman (including the Podman Machine host alias), Kubernetes and VM drivers carry it, in-process and as standalone binaries. The shared Kubernetes supervisor still does not relay. The relay now acknowledges OTLP metrics and logs exports with an empty success and discards them, so agent SDKs that export every signal to the generic endpoint stop retrying. Discards are counted and logged once. Local dev gateway tasks, the Skaffold values and the e2e harness enable the table; OPENSHELL_E2E_OTLP_AGENT=0 runs the inert case with trace export still on. Signed-off-by: Roland Huß <rhuss@redhat.com> * docs(supervisor): describe the agent trace relay architecture Add an "Agent traces" section to the architecture page and document the relay in the crate READMEs: the reserved-destination hook in the network supervisor, the OTEL defaults the sandbox runtime injects, and how the Docker, Podman, Kubernetes, and VM drivers pass the agent endpoint. Signed-off-by: Roland Huß <rhuss@redhat.com> * fix(sandbox): disable OTLP compression in the relay env and cover exec children The relay refuses compressed bodies, so an image or creation request that sets gzip compression without naming an endpoint received the relay default and lost every trace export to a 415. The relay default now also sets OTEL_EXPORTER_OTLP_COMPRESSION and OTEL_EXPORTER_OTLP_TRACES_COMPRESSION to none, as #4275 asks. Boundary exec children start from a cleared environment and never see the image ENV, yet the relay default was suppressed when the sandbox's own environment named an endpoint, which left exec children with no endpoint at all. apply_otlp_relay_env now takes a ChildEnvironment so only children that inherit the sandbox environment consult it. Add a probe test for both TLS environment modes together with the relay default, now that the mode-aware TLS layering lives in compose_canonical_process_environment, and describe the whole composition at its call sites. Signed-off-by: Roland Huß <rhuss@redhat.com> * fix(supervisor): flush the agent trace relay before suspending The suspend lifecycle from #4765 returned without shutting the relay down, so Drop discarded batches the relay had already acknowledged to the agent, which does not resend them after resume. Shut the relay down, bounded by FLUSH_TIMEOUT, once the workload is frozen and before networking is dropped. Also tidy what the rebases left behind: drop a stray blank line inside the upstream startup block, rename SandboxInstanceConfig::agent_otlp_endpoint to otlp_agent_endpoint to match every other identifier, serialize the OCSF table through the generic serialize_enabled, correct the UnsupportedScheme doc comment (the supervisor's own exporter does speak TLS), and build the reserved-destination proxy test identities with host-native paths. Signed-off-by: Roland Huß <rhuss@redhat.com> * test(e2e): scope the Podman OTLP default and restore upstream lockfiles The Podman wrapper enabled OTLP export and the agent relay for every e2e run, including single-target CI jobs such as additional_ca. Enable it only for the full suite or a selection that includes otlp_relay, following the SPIFFE fixture pattern; the Docker wrapper stays opt-in. The e2e and features lockfiles had downgraded libsqlite3-sys, crossbeam-queue and hybrid-array against upstream with no manifest change requiring it. Restore upstream's lockfiles and add only the packages the new e2e dependencies pull in. Signed-off-by: Roland Huß <rhuss@redhat.com> --------- Signed-off-by: Roland Huß <rhuss@redhat.com>
Sandbox runtime
openshell-sandbox owns the Linux workload isolation mechanisms and their kernel
qualification probes. The linux module contains seccomp notifications and child
filters, task-memory access, process-signal mediation, procfs descriptor lookup,
socket identity, Landlock qualification, and the workload launch thread.
sandbox::linux composes policy enforcement. The top-level linux module owns
the lower-level mechanisms used by that orchestration and the boundary server.
Unsafe kernel ABI, process-launch, and descriptor-ownership operations stay in
this runtime crate behind safe APIs where their contracts can be enforced.
The network broker retains source socket descriptors only while preparing a
workload endpoint. A successful connection or listen releases that reference;
the kernel then tracks the lifetime through workload-owned descriptors. Native
accept and repeated listen preserve the socket's loopback confinement. Closing
the last workload reference to a listener removes it without waiting for registry
collection. Broker source descriptors always use close-on-exec independently of
the workload's requested descriptor flags.
A boundary configured with RuntimeInitiatedTlsTcp binds no listener. It dials
only the driver-provided supervisor addresses, which must be concrete,
non-loopback, and share one port, and serves TLS and gRPC over that outbound
connection. It keeps one spare connection that the supervisor has not yet used
and dials a replacement when the supervisor starts HTTP/2 on it or the spare
closes. Live connections keep the same disconnect, freeze, and recovery-deadline
handling as listener mode, and the workload still cannot connect to the control
port.
A runtime of a supervisor shared by several sandboxes (one with a route handle) names its route in a version 2 preface on every dial and follows the supervisors' redirects, on the same port and never on loopback, at most two hops. A supervisor that closes the negotiation without replying, as one from before version 2 does, is dialed again with a version 1 preface and treated as a lone supervisor; this changes only routing, since TLS pinning, request authentication, and takeover grants are unaffected. It keeps a spare at the member whose connection it last confirmed, its authenticated owner; an unauthenticated accept never makes a member the owner. Once a supervisor names itself a pool member or redirects it, the runtime also dials the configured addresses past every known member every 15 seconds, up to three times, so a member that hosts the route without a spare gets one: a previous owner that stays reachable after the gateway granted the route elsewhere, holding an unused spare, cannot keep the new owner from the runtime. A lone shared supervisor never names itself, so it holds the one spare and is never probed. The runtime keeps at most three unused spares, one per member, and never closes an established connection to refresh routing. A member's address grants nothing: a later term at a fenced owner's address authenticates with its own grant. A retired connection that does not shut down gracefully within two seconds, such as an unused spare, has its transport dropped.
When AgentSpec::otlp_relay_active is set, child_env::apply_otlp_relay_env
gives every workload process (main, exec, and SSH) OTEL_EXPORTER_OTLP_ENDPOINT
pointing at the relay address, OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf, and
none for OTEL_EXPORTER_OTLP_COMPRESSION and
OTEL_EXPORTER_OTLP_TRACES_COMPRESSION, and removes any inherited
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT. These defaults apply after the provider
variables. If the creation request or a per-exec environment names a generic
or traces-specific OTLP endpoint, the runtime injects nothing, and the
workload exports to that endpoint under the sandbox's network policy. An
endpoint in the image ENV does the same for the main process; exec children
start from a cleared environment (ChildEnvironment::Cleared), never see the
image's endpoint, and still get the relay default.
Supervisor takeover
The boundary advertises openshell.sandbox.supervisor-takeover. A replacement
supervisor process may attach to a running generation only with a
gateway-signed takeover grant for a later credential epoch, its bearer of that
epoch, and the grant's supervisor process identity. The attach is staged and
promoted at confirmation, which advances the accepted epoch, pins the new
process, and closes every connection of the previous owner; credentials of
earlier epochs are refused from then on. Refused attempts emit an OCSF
configuration-state event and a sandbox-supervisor-takeover-denied finding.
Attach and confirmation are serialized, and the registry checks an attach before it binds anything. Owner operations (exec, signals, termination, provider updates, network decisions, DNS answers) hold an ownership guard from authorization through their effect, and a promotion takes it exclusively, so none of the replaced owner's operations takes effect afterwards. The guard is asynchronous: tasks on the runtime's workers wait for it without blocking them. A confirmation replay on the connection in force promotes nothing and skips the guard. Promotion and the supervisory state change together, so a replacement lost mid-promotion leaves the workload frozen under a fresh recovery deadline. Each supervisor offers a recovery deadline when it attaches (10 to 300 seconds; the runtime refuses any other before binding anything), and the deadline of the connection in force when supervision is lost bounds the freeze: 30 seconds without an offer. A supervisor pool offers 60 seconds unless the gateway configures another.
The runtime keeps the trust-on-first-use executable pins of the generation
(binary_pins): the first digest it offers for each path, from its own
measurements. A contradicting identity is offered as a resolution failure,
which every supervisor denies, so a replacement with an empty cache cannot pin
a binary the workload replaced.