krishicks
8091f66877
Since #2726 the canonical main process's stdout and stderr are captured in pipes that feed only the in-memory replay buffer used by sandbox connect. Agent output therefore never reaches the container's own stdout and stderr, so it is missing from kubectl logs, docker logs, and podman logs and from anything that collects container logs. Before #2726 the entrypoint inherited the container's descriptors and its output appeared there. Copy the main process's output to the launcher's stdout and stderr in addition to the replay buffer, restoring the earlier behavior: - Output is copied byte for byte to the matching stream from a forwarder thread per stream, after it is published to the replay buffer. When the container runtime falls behind on a stream, that stream's reader waits instead of dropping output, so backpressure reaches the agent as it did with inherited descriptors, while the other stream and attachments keep receiving output. - Before the main process's exit is published, the output readers finish and queued output is drained to the container log, so an agent's final lines are not lost at shutdown. A 30 second deadline covers both; when it expires, readers waiting on the container log are released and drain the pipes into the replay buffer only, so a stalled container log cannot block exit reporting. - PTY-mode processes are not copied. The terminal stream carries escape sequences and echoed input, and terminal commands never reached the container log before #2726. - Exec, SSH, and SFTP sessions are not copied. Launcher log lines keep their existing format and remain in the container's stderr. They are written as whole lines, and a newline is inserted first when the agent left stderr mid-line, so launcher and agent lines do not merge. The Docker and VM drivers appended the tail of the workload's output to failure messages: Docker the workload container's log, and the VM driver the guest console, which carries the launcher's stdout and stderr. Those messages land in the sandbox's Ready condition and in platform events that the gateway republishes to the sandbox event stream. With agent output in that log, those messages would carry arbitrary agent output, including anything sensitive the agent prints, into gateway status and events. The supervisor starts its health endpoint only after the agent starts, so every Docker failure path could include agent output, and the VM driver reports one whenever the VM or host supervisor exits. Forward only the supervisor's log tail, matching the Podman driver, which reads the workload log solely to match fixed launcher markers and never forwards raw workload output. The workload's output remains available through docker logs and the VM's rootfs-console.log. Document where main process output appears in the logging docs and the cluster debugging skill. Closes #3928 Signed-off-by: Kris Hicks <khicks@nvidia.com>
openshell-driver-docker
Docker-backed compute driver for local and remote OpenShell gateways.
The driver uses bollard to manage sandbox resources through the configured
Docker API socket. When socket_path is unset, it selects the first standard
local socket that responds to an API ping. An explicitly selected Docker driver
falls back to /var/run/docker.sock when no candidate responds.
When the gateway configures [openshell.gateway.otlp], the in-process driver
exports spans to the same OTLP/gRPC collector as
openshell-driver-docker. The standalone driver accepts
OPENSHELL_OTLP_ENDPOINT, continues W3C trace context from gateway RPC
metadata, and flushes spans during graceful shutdown.
Runtime Model
Caller driver config is disabled by default. Existing volumes require administrator-controlled approval labels; raw bind mounts have no supported label resolver and are denied under enforcement. GPU devices are temporarily exempt. Inspect labels again before launch, restart, and during reconciliation. See resource admission configuration.
The driver creates two containers for each sandbox:
openshell-sandboxis PID 1 in the workload container. It owns the workload process tree, seccomp notification broker, mandatory Landlock baseline, binary identity, exec/signal/wait/PTY operations, and loopback forwarding.openshell-supervisorruns in a separate companion container. It owns the gateway session, policy engine, credentials, interception CA, SSH relay, L7 inspection, DNS policy, and external upstream connections.
Both containers are non-root, request no capabilities, and set no-new-privileges. A shared named volume carries the authenticated Unix socket and sandbox bootstrap material. A second supervisor-only volume carries the supervisor JWT and gateway client credentials and is never mounted into the workload.
The workload uses network_mode=none. Its seccomp user-notification broker
mediates every supported TCP and DNS operation, attributes it to the calling
binary, and sends the request across the private channel. The supervisor
authorizes the request before it opens an upstream connection. Docker's absent
workload network is the mandatory outer fence if mediation fails or is
bypassed. The trusted supervisor companion uses Docker host networking, where
it reaches the gateway's primary loopback listener and originates approved
egress.
The driver copies trusted runtime bytes from the configured supervisor image through the Docker archive API. No workload launch depends on a host bind mount or a tool supplied by the workload image, so the same path works with local, remote, and VM-backed Docker daemons.
Corporate Proxy Egress
[openshell.drivers.docker] accepts the operator-owned corporate proxy fields
https_proxy, no_proxy, proxy_auth_file,
proxy_auth_allow_insecure, proxy_connect_by_hostname, and
proxy_ca_bundle. Sandbox environment, image contents, and per-sandbox driver
configuration cannot select or override them.
The driver validates the proxy URL and cross-field relationships at gateway
startup. It bounded-reads proxy credentials and the PEM CA bundle before use.
Missing, unreadable, empty, oversized, malformed, or certificate-free CA files
fail closed. A CA bundle requires https_proxy, although the proxy URL may be
http:// when the proxy intercepts destination TLS.
For each supervisor launch, Docker copies the credential and CA contents into
the existing supervisor-only named volume. It passes only the fixed container
paths /.openshell/supervisor/upstream-proxy-auth and
/.openshell/supervisor/upstream-proxy-ca-bundle.pem to the supervisor. It
does not bind-mount the gateway-host files, which preserves remote-daemon
support and keeps host paths out of workload container metadata.
The supervisor trusts the configured CA for its TLS connection to an HTTPS proxy and for destination certificates re-signed by a TLS-intercepting proxy. It also includes the corporate root in the generated combined trust bundle used by workload processes.
Identity and Workspace
Before creating the workload, the driver pins the image ID and reads its
passwd/group databases through a stopped metadata container. It resolves the
admitted policy identity, or the image Config.User fallback, into one exact
non-root UID, primary GID, and supplementary-group set. Docker launches
openshell-sandbox with that identity, and the sandbox uses the same identity
for every canonical and exec process. UID or GID zero and unresolved symbolic
identities are rejected.
An absolute OCI working directory becomes the workspace. An empty, root (/),
or explicit /sandbox declaration uses /sandbox. Any other workdir must
already exist without symlink components. The resolved identity must be able to
traverse every parent and write and enter the workdir; OpenShell does not
change its ownership or mode.
Image VOLUME declarations and user mounts must not cover the workdir, one of
its parents, or the reserved /.openshell runtime/channel tree. OpenShell asks
the kernel to validate access under the final identity, so POSIX ACL and host
LSM decisions remain authoritative.
Container Contract
| Setting | Purpose |
|---|---|
Exact non-root user and group_add |
Gives sandbox and workload the same immutable UID/GID/group identity required for capability-free observation. |
cap_drop = ALL, no cap_add, no-new-privileges |
Prevents either container from acquiring Linux capabilities. |
| Docker default seccomp and AppArmor profiles | Retains runtime hardening; startup confirmation fails closed if nested seccomp notification is unavailable. |
network_mode = none on the workload |
Removes direct external routes. |
network_mode = host on the supervisor |
Lets the trusted supervisor reach the gateway's primary loopback listener and originate approved upstream connections. |
restart_policy = no |
Keeps canonical main-process exit terminal. |
PidsLimit |
Applies the configured sandbox PID budget. Omit sandbox_pids_limit to use OpenShell's default. Explicit zero is invalid. |
| Private named volumes | One carries the authenticated sandbox/supervisor channel. The other is mounted only into the supervisor and contains its JWT and private gateway credentials. |
In-memory /run/openshell-supervisor-ca tmpfs |
Holds only the public supervisor CA certificate and trust bundle without making all of /run writable. |
| CDI GPU request | Assigns the exact validated CDI devices requested by driver config or count-based selection. |
Stop, Start, and Delete
Stop terminates the supervisor companion and stops the workload container without removing it. Docker retains the workload writable layer and attached volumes. Start stages a fresh sandbox bootstrap bundle, restarts that workload, and creates a new supervisor companion. A durably stopped sandbox stays stopped across gateway restarts.
Delete force-removes both containers, the driver-owned runtime volumes, and the host-private runtime descriptor. Missing or altered descriptor and channel resources fail closed; the driver does not run an older combined-supervisor layout.
Driver Config Mounts
The gateway forwards the docker block from --driver-config-json. Supported
mount types are:
bind: an absolute daemon-host path, allowed only when[openshell.drivers.docker].enable_bind_mounts = trueand label admission is explicitly disabled.volume: an existing named volume. The driver never creates or removes a user-supplied volume. Bind-backed local volumes requireenable_bind_mounts = true.tmpfs: an in-memory filesystem with optional size and mode.
Host bind mounts are disabled by default because they expose daemon-host paths to sandbox requests. User bind and volume mounts are read-only by default. Targets must be absolute, normalized paths and cannot overlap the workspace root or OpenShell control paths.
Example:
docker volume create openshell-work
openshell sandbox create \
--driver-config-json '{"docker":{"mounts":[{"type":"volume","source":"openshell-work","target":"/sandbox/work"}]}}' \
-- claude
Runtime Image
sandbox_runtime_image contains the statically linked musl
/openshell-sandbox binary. The driver extracts that binary as bytes and
stages it into the stopped workload. supervisor_image contains the
dynamically linked glibc /openshell-supervisor binary that runs in the
host-networked supervisor container. Release and gateway image builds bake
matching image tags into the binary.
Gateway session and TLS
OPENSHELL_ENDPOINT and gateway authentication material are injected only into
the supervisor companion. The workload never receives the sandbox JWT, gateway
client TLS key, policy authority, or interception CA private key.
When no endpoint is configured, the supervisor connects to
127.0.0.1:<gateway-port>. Set grpc_endpoint when the gateway is not on the
Docker daemon host. A configured HTTPS server certificate must include the
endpoint host in its subject alternative names.
The driver publishes host loopback as the backend address for
host.openshell.internal. Policy DNS resolves that reserved name through the
mediated path, so policies can reach host services without a Docker bridge,
container DNS alias, or another gateway listener.
For HTTPS endpoints, the supervisor receives only the gateway CA and uses its sandbox-scoped bearer token to authenticate RPCs. User client certificates and private keys are not delivered to either container.
Docker Engine on Linux supports host networking directly. Docker Desktop requires host networking to be enabled in Settings and does not support it when Enhanced Container Isolation is enabled.
The supervisor owns these security-critical variables:
OPENSHELL_ENDPOINTOPENSHELL_SANDBOX_IDOPENSHELL_SANDBOXOPENSHELL_SANDBOX_TOKEN_FILEOPENSHELL_SSH_SOCKET_PATHOPENSHELL_MAIN_PROCESS_SPECOPENSHELL_TLS_CAwhen HTTPS is enabled
Template and sandbox environment is encoded in the protected bootstrap and exposed only to workload children. Workload input cannot override security-critical supervisor variables.