Files
OpenShell/crates/openshell-driver-docker
Roland Huss fdce6ef260 feat(supervisor): relay agent OTLP traces to the collector (#4276)
* feat(supervisor): relay agent OTLP traces to the collector

Serve the reserved endpoint 192.0.0.8:4318 inside the supervisor, attribute each batch with openshell.sandbox.id and openshell.telemetry.source=agent at the protobuf wire level, buffer it with fixed bounds, and forward it over OTLP/gRPC to the collector the supervisor already uses, flushing before the exit report and keeping the relay up for exec sessions.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* refactor(supervisor): let the OTLP relay exporter take a prepared channel

Expose Exporter::with_channel as the seam for a collector transport built once in openshell-otel and shared by the gateway, the supervisor's own spans, and the relay. Exporter::new keeps building the plaintext channel for now.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* feat(sandbox): leave caller-set OTLP endpoints alone

Inject the relay variables only when the sandbox creation request names no OTLP endpoint, generic or traces-specific. A caller who names one has chosen where the agent exports; overwriting it would silently redirect an agent that deliberately exports elsewhere. A protocol or compression setting alone does not suppress injection. Applies to both spawn paths and to exec children, with the docs updated to match.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* fix(supervisor): address OTLP relay review feedback

Inject only the generic OTEL_EXPORTER_OTLP_ENDPOINT and
OTEL_EXPORTER_OTLP_PROTOCOL as lowest-precedence defaults, and only
while the relay is active. The supervisor now starts the relay before
attach and signals it to the sandbox runtime through
AgentSpec::otlp_relay_active. An endpoint in the creation request, the
image ENV, or an exec environment suppresses the defaults, and a
provider-supplied traces endpoint is removed.

Harden the relay itself:
- skip the exit flush and shutdown wait when the last export failed
- count batches still pending on Drop as discarded and log them
- validate the collector endpoint with the shared openshell-otel helper
- accept Content-Encoding: identity
- fix the 413 and shutdown 503 messages and the memory bound docs
- narrow the attribution claim to resource attributes

Document best-effort delivery and the 404 for metrics and logs.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* feat(supervisor): gate the agent trace relay behind [otlp.agent]

The relay started whenever a sandbox supervisor had a trace endpoint, so
enabling gateway trace export also changed every workload's environment.
Make it opt-in: `[openshell.gateway.otlp.agent]` with `enabled = true`
turns it on, and its optional `endpoint` falls back to the shared
`[openshell.gateway.otlp]` endpoint, like the OCSF table.

The gateway validates the endpoint and drivers pass it to supervisors as
OPENSHELL_OTLP_AGENT_ENDPOINT, separate from OPENSHELL_OTLP_ENDPOINT.
Docker, Podman (including the Podman Machine host alias), Kubernetes and
VM drivers carry it, in-process and as standalone binaries. The shared
Kubernetes supervisor still does not relay.

The relay now acknowledges OTLP metrics and logs exports with an empty
success and discards them, so agent SDKs that export every signal to the
generic endpoint stop retrying. Discards are counted and logged once.

Local dev gateway tasks, the Skaffold values and the e2e harness enable
the table; OPENSHELL_E2E_OTLP_AGENT=0 runs the inert case with trace
export still on.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* docs(supervisor): describe the agent trace relay architecture

Add an "Agent traces" section to the architecture page and document the
relay in the crate READMEs: the reserved-destination hook in the network
supervisor, the OTEL defaults the sandbox runtime injects, and how the
Docker, Podman, Kubernetes, and VM drivers pass the agent endpoint.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* fix(sandbox): disable OTLP compression in the relay env and cover exec children

The relay refuses compressed bodies, so an image or creation request that
sets gzip compression without naming an endpoint received the relay default
and lost every trace export to a 415. The relay default now also sets
OTEL_EXPORTER_OTLP_COMPRESSION and OTEL_EXPORTER_OTLP_TRACES_COMPRESSION to
none, as #4275 asks.

Boundary exec children start from a cleared environment and never see the
image ENV, yet the relay default was suppressed when the sandbox's own
environment named an endpoint, which left exec children with no endpoint at
all. apply_otlp_relay_env now takes a ChildEnvironment so only children that
inherit the sandbox environment consult it.

Add a probe test for both TLS environment modes together with the relay
default, now that the mode-aware TLS layering lives in
compose_canonical_process_environment, and describe the whole composition
at its call sites.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* fix(supervisor): flush the agent trace relay before suspending

The suspend lifecycle from #4765 returned without shutting the relay down,
so Drop discarded batches the relay had already acknowledged to the agent,
which does not resend them after resume. Shut the relay down, bounded by
FLUSH_TIMEOUT, once the workload is frozen and before networking is
dropped.

Also tidy what the rebases left behind: drop a stray blank line inside the
upstream startup block, rename SandboxInstanceConfig::agent_otlp_endpoint
to otlp_agent_endpoint to match every other identifier, serialize the OCSF
table through the generic serialize_enabled, correct the UnsupportedScheme
doc comment (the supervisor's own exporter does speak TLS), and build the
reserved-destination proxy test identities with host-native paths.

Signed-off-by: Roland Huß <rhuss@redhat.com>

* test(e2e): scope the Podman OTLP default and restore upstream lockfiles

The Podman wrapper enabled OTLP export and the agent relay for every e2e
run, including single-target CI jobs such as additional_ca. Enable it only
for the full suite or a selection that includes otlp_relay, following the
SPIFFE fixture pattern; the Docker wrapper stays opt-in.

The e2e and features lockfiles had downgraded libsqlite3-sys,
crossbeam-queue and hybrid-array against upstream with no manifest change
requiring it. Restore upstream's lockfiles and add only the packages the
new e2e dependencies pull in.

Signed-off-by: Roland Huß <rhuss@redhat.com>

---------

Signed-off-by: Roland Huß <rhuss@redhat.com>
2026-10-28 15:26:52 +00:00
..

openshell-driver-docker

Docker-backed compute driver for local and remote OpenShell gateways.

The driver uses bollard to manage sandbox resources through the configured Docker API socket. When socket_path is unset, it selects the first standard local socket that responds to an API ping. An explicitly selected Docker driver falls back to /var/run/docker.sock when no candidate responds.

When the gateway configures [openshell.gateway.otlp], the in-process driver exports spans to the same OTLP/gRPC collector as openshell-driver-docker. The standalone driver accepts OPENSHELL_OTLP_ENDPOINT, continues W3C trace context from gateway RPC metadata, and flushes spans during graceful shutdown. When [openshell.gateway.otlp.agent] is enabled, the driver also sets OPENSHELL_OTLP_AGENT_ENDPOINT on the supervisor container, which starts the agent trace relay. The standalone driver takes that endpoint as --otlp-agent-endpoint.

Runtime Model

Caller driver config is disabled by default. Existing volumes require administrator-controlled approval labels; raw bind mounts have no supported label resolver and are denied under enforcement. GPU devices are temporarily exempt. Inspect labels again before launch, restart, and during reconciliation. See resource admission configuration.

The driver creates two containers for each sandbox:

  • openshell-sandbox is PID 1 in the workload container. It owns the workload process tree, seccomp notification broker, mandatory Landlock baseline, binary identity, exec/signal/wait/PTY operations, and loopback forwarding.
  • openshell-supervisor runs in a separate companion container. It owns the gateway session, policy engine, credentials, interception CA, SSH relay, L7 inspection, DNS policy, and external upstream connections.

Both containers are non-root, request no capabilities, and set no-new-privileges. A shared named volume carries the authenticated Unix socket and sandbox bootstrap material. A second supervisor-only volume carries the supervisor JWT and gateway client credentials and is never mounted into the workload.

The workload uses network_mode=none. Its seccomp user-notification broker mediates every supported TCP and DNS operation, attributes it to the calling binary, and sends the request across the private channel. The supervisor authorizes the request before it opens an upstream connection. Docker's absent workload network is the mandatory outer fence if mediation fails or is bypassed. The trusted supervisor companion originates approved egress. Its operator-owned supervisor_network_mode defaults to host. Set bridge to run the supervisor on Docker's bridge network; auto also selects host.

The driver copies trusted runtime bytes from the configured supervisor image through the Docker archive API. No workload launch depends on a host bind mount or a tool supplied by the workload image, so the same path works with local, remote, and VM-backed Docker daemons.

Corporate Proxy Egress

[openshell.drivers.docker] accepts the operator-owned corporate proxy fields https_proxy, no_proxy, proxy_auth_file, proxy_auth_allow_insecure, proxy_connect_by_hostname, and proxy_ca_bundle. Sandbox environment, image contents, and per-sandbox driver configuration cannot select or override them.

The driver validates the proxy URL and cross-field relationships at gateway startup. It bounded-reads proxy credentials and the PEM CA bundle before use. Missing, unreadable, empty, oversized, malformed, or certificate-free CA files fail closed. A CA bundle requires https_proxy, although the proxy URL may be http:// when the proxy intercepts destination TLS.

For each supervisor launch, Docker copies the credential and CA contents into the existing supervisor-only named volume. It passes only the fixed container paths /.openshell/supervisor/upstream-proxy-auth and /.openshell/supervisor/upstream-proxy-ca-bundle.pem to the supervisor. It does not bind-mount the gateway-host files, which preserves remote-daemon support and keeps host paths out of workload container metadata.

The supervisor trusts the configured CA for its TLS connection to an HTTPS proxy and for destination certificates re-signed by a TLS-intercepting proxy. It also includes the corporate root in the generated combined trust bundle used by workload processes.

Identity and Workspace

Before creating the workload, the driver pins the image ID and reads its passwd/group databases through a stopped metadata container. It resolves the admitted policy identity, or the image Config.User fallback, into one exact non-root UID, primary GID, and supplementary-group set. Docker launches openshell-sandbox with that identity, and the sandbox uses the same identity for every canonical and exec process. UID or GID zero and unresolved symbolic identities are rejected.

An absolute OCI working directory becomes the workspace. An empty, root (/), or explicit /sandbox declaration uses /sandbox. Any other workdir must already exist without symlink components. The resolved identity must be able to traverse every parent and write and enter the workdir; OpenShell does not change its ownership or mode.

Image VOLUME declarations and user mounts must not cover the workdir, one of its parents, or the reserved /.openshell runtime/channel tree. OpenShell asks the kernel to validate access under the final identity, so POSIX ACL and host LSM decisions remain authoritative.

Container Contract

Setting Purpose
Exact non-root user and group_add Gives sandbox and workload the same immutable UID/GID/group identity required for capability-free observation.
cap_drop = ALL, no cap_add, no-new-privileges Prevents either container from acquiring Linux capabilities.
Docker default seccomp and AppArmor profiles Retains runtime hardening; startup confirmation fails closed if nested seccomp notification is unavailable.
network_mode = none on the workload Removes direct external routes.
network_mode = host on the supervisor Lets the trusted supervisor reach the gateway's primary loopback listener and originate approved upstream connections.
restart_policy = no Keeps canonical main-process exit terminal.
PidsLimit Applies the configured sandbox PID budget. Omit sandbox_pids_limit to use OpenShell's default. Explicit zero is invalid.
Private named volumes Separate volumes carry sandbox bootstrap keys, the runtime-initiated Unix socket, and supervisor-only credentials. The supervisor mounts bootstrap keys read-only and only the socket volume writable.
In-memory /run/openshell-supervisor-ca tmpfs Holds only the public supervisor CA certificate and trust bundle without making all of /run writable.
CDI GPU request Assigns the exact validated CDI devices requested by driver config or count-based selection.

New containers dial the supervisor at /.openshell/channel/control/supervisor/control.sock. The runtime remains the TLS/gRPC server; the supervisor verifies its pinned identity and supplies the existing bearer credential. Saved descriptors retain the legacy Unix listener for existing containers, including across restart. New generations require matching supervisor and sandbox runtime binaries.

Stop, Start, and Delete

Stop terminates the supervisor companion and stops the workload container without removing it. Docker retains the workload writable layer and attached volumes. Start stages a fresh sandbox bootstrap bundle, restarts that workload, and creates a new supervisor companion. A durably stopped sandbox stays stopped across gateway restarts.

Delete force-removes both containers, the driver-owned runtime volumes, and the host-private runtime descriptor. Missing or altered descriptor and channel resources fail closed; the driver does not run an older combined-supervisor layout.

Driver Config Mounts

The gateway forwards the docker block from --driver-config-json. Supported mount types are:

  • bind: an absolute daemon-host path, allowed only when [openshell.drivers.docker].enable_bind_mounts = true and label admission is explicitly disabled.
  • volume: an existing named volume. The driver never creates or removes a user-supplied volume. Bind-backed local volumes require enable_bind_mounts = true.
  • tmpfs: an in-memory filesystem with optional size and mode.

Host bind mounts are disabled by default because they expose daemon-host paths to sandbox requests. User bind and volume mounts are read-only by default. Targets must be absolute, normalized paths and cannot overlap the workspace root or OpenShell control paths.

Example:

docker volume create openshell-work

openshell sandbox create \
  --driver-config-json '{"docker":{"mounts":[{"type":"volume","source":"openshell-work","target":"/sandbox/work"}]}}' \
  -- claude

Runtime Image

sandbox_runtime_image contains the statically linked musl /openshell-sandbox binary. The driver extracts that binary as bytes and stages it into the stopped workload. supervisor_image contains the dynamically linked glibc /openshell-supervisor binary that runs in the supervisor companion container. Release and gateway image builds bake matching image tags into the binary.

Gateway session and TLS

OPENSHELL_ENDPOINT and gateway authentication material are injected only into the supervisor companion. The workload never receives the sandbox JWT, gateway client TLS key, policy authority, or interception CA private key.

When no endpoint is configured, the supervisor connects to 127.0.0.1:<gateway-port>. Set grpc_endpoint when the gateway is not on the Docker daemon host. A configured HTTPS server certificate must include the endpoint host in its subject alternative names.

In host mode, the driver publishes host loopback as the backend address for host.openshell.internal. Policy DNS resolves that reserved name through the mediated path, so policies can reach host services without a Docker bridge, container DNS alias, or another gateway listener.

For HTTPS endpoints, the supervisor receives only the gateway CA and uses its sandbox-scoped bearer token to authenticate RPCs. User client certificates and private keys are not delivered to either container.

Host mode on Docker Desktop requires host networking enabled in Settings. For bridge mode or a gateway endpoint using host.docker.internal, the driver resolves Docker's host-gateway once during initialization and caches it. A temporary container runs --help from the immutable supervisor image with no network, no capabilities, no privilege escalation, and a read-only root filesystem. The driver downloads Docker's generated /etc/hosts through the archive API, preferring a unique IPv4 address and otherwise a unique IPv6 address. Resolution has a 30-second timeout; forced cleanup has a separate 10-second timeout and runs after success or failure.

The driver uses the selected address for both host aliases and the runtime descriptor's trust pin; workload DNS and hosts files cannot select it. Host mode with other gateway endpoints retains its gateway-address pin. An explicit operator-owned host_gateway_ip overrides discovery and skips the temporary container. No supervisor changes or per-sandbox discovery are required.

When grpc_endpoint is omitted, bridge mode uses host.docker.internal on Desktop or for a wildcard gateway listener, and the concrete bind address for other non-loopback Linux listeners. The port is the gateway bind port; remapped container publications require an explicit endpoint. Native Linux loopback-only listeners cannot serve a bridge supervisor and produce an actionable error. Explicit endpoints retain TLS verification and are never rewritten; bridge rejects loopback/unspecified endpoint addresses. Host services bound only to loopback are also unreachable from a native Linux bridge supervisor.

The selected network mode does not change the sandbox's mandatory Landlock and seccomp qualification.

The supervisor owns these security-critical variables:

  • OPENSHELL_ENDPOINT
  • OPENSHELL_SANDBOX_ID
  • OPENSHELL_SANDBOX
  • OPENSHELL_SANDBOX_TOKEN_FILE
  • OPENSHELL_SSH_SOCKET_PATH
  • OPENSHELL_MAIN_PROCESS_SPEC
  • OPENSHELL_TLS_CA when HTTPS is enabled

Template and sandbox environment is encoded in the protected bootstrap and exposed only to workload children. Workload input cannot override security-critical supervisor variables.