* feat(network): support additional destination CAs Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): harden additional CA propagation Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): narrow additional CA review scope Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): harden additional CA rotation Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * feat(network): complete additional CA delivery Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(docker): remove stale runtime fixture field Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): adapt additional CA series to upstream main Resolve rebase fallout against current upstream main after the series was rebased 214 commits forward: - follow upstream #3110: supervisor→gateway TLS is CA-only, so configure_gateway_tls and its isolation tests drop the client identity - follow upstream tls_secret_names reduction to [String; 1] in the Podman driver while keeping the network trust bundle delivery - port [openshell.supervisor.network] rendering into the openshell.gatewayConfigToml serializer in _toml.tpl - adapt tests to upstream struct/function changes (DriverStartupContext, SandboxRunConfig, IsolationSpecInput, supervisor_pod, VM bootstrap image) - drop conformance-artifact-prefix plumbing removed upstream Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(network): adopt upstream additional CA bundle surface Unify the additional destination CA series on the mechanism merged to main in #3996: one operator pre-assembled PEM bundle, the --additional-ca-bundle supervisor flag, and the /etc/openshell/tls/additional/ca-bundle.pem staging path. - Replace the gateway-level additional_ca_cert_paths list with a single additional_ca_bundle file validated and staged at gateway startup. - Restore the upstream supervisor argument path and fold-into-roots behavior; drop the --network-additional-ca-bundle and --network-additional-ca-digest contracts. - Keep gateway normalization bounds, content-addressed staging, digest-based driver reconciliation, and stop/start rotation; the freshness check now runs at the gateway/driver boundary instead of the supervisor boundary. - Reject additional_ca_bundle in operator driver JSON config; standalone drivers self-source the bundle from --sandbox-additional-ca-bundle with the same validation. - Extend the Kubernetes ConfigMap name digest segment to keep the full 63-byte budget after the prefix rename. - Box the persisted-sandbox reconciliation futures so callers stay under the clippy::large_futures threshold. Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and tests, e2e wiring, gateway configuration docs, and the cluster debugging skill to the single mechanism. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(ci): make gateway startup tests wrap-tolerant and refresh example lockfile The network trust startup tests asserted on raw substrings of gateway diagnostics, but the CLI renderer wraps output at a fixed width with continuation markers and hard-breaks long paths. Compare assertions on whitespace- and continuation-squashed text so they hold regardless of wrap position. Format the e2e workspace (separate cargo workspace, not covered by the root cargo fmt run) and refresh examples/isolation-backend-v8/Cargo.lock for the libc dependency added to the network trust crate. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(podman): reuse TLS CA secret during stopped-pair trust replacement The additional CA e2e never ran in CI before the first maintainer test dispatch, and its local container inspection was broken in two ways: it located the supervisor container through e2e network membership (the supervisor is host-networked or runtime-bridged, so the filter matched the CI job container instead), and it required the gateway mTLS CA as a bind mount (Podman delivers it as a container secret when user namespaces remap UIDs). Select the supervisor by managed labels and accept the secret delivery form. The lifecycle leg then exposed a driver bug: stopped-pair trust replacement re-created the per-sandbox TLS CA secret, colliding with the secret still owned by the renamed stopped pair. Mirror the token and resolver secret handling: require and reuse the existing secret during replacement instead of creating it. Validated end-to-end with the Podman additional CA e2e lane. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(server): restore upstream recovery semantics for start-request failures Recovery no longer marks sandboxes Error/InvalidPersistedSandbox when driver_start_request fails during the startup or lifecycle sweeps. Gateway configuration problems (for example a missing launch signer) must not permanently fail unrelated sandbox records; the sweep logs a warning and skips them, matching the behavior before the additional CA work. Sandbox-record failures surface through the normal start path. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(podman): refresh stale supervisors without replacing workloads Replace the stopped-pair trust reconciliation with supervisor-only recreation. The destination-trust generation label moves from the workload to the supervisor container, and start_sandbox recreates only the supervisor when its generation is missing or stale. The workload container, its volumes, its secrets, and its workspace data are never replaced: the additional CA bundle is mounted only into the supervisor. Running sandboxes are never gated on trust labels, and recovery without fresh launch credentials keeps the existing supervisor. The intentional -removal fence machinery in the watcher existed only for workload replacement and is removed with it. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(docker): drop workload replacement for destination-trust changes Delete the stopped-workload trust reconciliation path and all of its machinery: the trust-generation label on workload containers, the generation match check in the start fence, the stopped-sandbox replacement, the bounded workspace archive copy with its 512 MiB cap and regular-file-only validation, and the boundary-state rollback helpers. No replacement machinery is needed: the trusted supervisor container is already re-created from the gateway's current configuration on every start via ensure_control_process_for_container, so stop/start picks up the current additional CA bundle while the workload container, its volumes, and its workspace data are never replaced. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * docs(configuration): drop reconciliation wording from additional CA lifecycle The stop/start lifecycle description already matches the supervisor-only refresh model; remove the two remaining references to restart reconciliation of workload containers. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): port additional CA test fixtures to the crypto interface Route the gateway TLS isolation test PKI, the destination CA source fixtures, and the gateway startup trust tests through openshell_crypto::pki now that rcgen access is centralized there: generate_keypair/self_signed/signed_by replace the direct rcgen calls and CertifiedKey field access follows the wrapper's shape. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(e2e): repair additional CA lane for Docker and non-container builds The first maintainer test dispatch revealed two latent issues in the additional CA e2e that no local run had exercised: the Podman gateway CA secret helper referenced ContainerEngine without the container feature gate, breaking Kubernetes and VM builds of the test, and the Docker branch asserted a client ca.crt bind mount that the Docker driver never creates. Gate the helper like its callers, and assert Docker's actual delivery path instead: the gateway mTLS CA staged as tls/ca.pem inside the read-only supervisor state volume, selected through the OPENSHELL_TLS_CA environment pointer. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(helm): restore alphabetical README ordering for supervisor keys The helm:docs check regenerates the chart README with helm-docs, which orders rows alphabetically; a conflict resolution had placed supervisor.network.additionalCaConfigMapName after the supervisor.podSecurityContext rows. Regenerate with mise run helm:docs. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * chore: refresh testkit lockfile for openshell-core libc dependency Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(driver-podman): stage operator CA bundle as driver-owned artifact Standalone Podman drivers accepting --sandbox-additional-ca-bundle previously kept the operator's source path as the trust artifact path. The immutable-artifact verification performed on every create/start requires exactly mode 0444, which an operator-owned source path cannot guarantee, so a readable-but-writable source failed reconciliation. Read the operator bundle with the same bounded, no-symlink validation, then stage the normalized material into a driver-owned artifact inside a private tempdir held for the driver process lifetime. The artifact is created 0600, synced, and only then restricted to 0444 so no observer ever sees a writable artifact. The operator source and its permissions are never referenced after startup. Addresses GATOR-291f43b2-02. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(driver-podman): pin retained workload image during supervisor refresh Supervisor-only trust reconciliation re-entered create_sandbox_with_existing_token, which re-pulled and re-resolved the workload image tag even though the workload container is retained. If the tag moved to a different image between stop and start, identity resolution and child-environment construction would reflect the new image while the retained workload still runs the original one, leaving mode-0600 bootstrap files the retained user cannot read. During a refresh with a retained workload, inspect the stopped container and pin to the immutable image ID it was created from (Config.Image); never pull or resolve the request tag. The requested reference is still used for child metadata so the retained workload's environment is byte-identical to its original launch. Addresses GATOR-291f43b2-01. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * test(e2e): verify host-only VM trust delivery and drop curl dependency The additional-CA lane's VM assertions read the destination CA out of the guest overlay and a guest-side supervisor-argv marker — artifacts from the guest-supervisor design that no longer exist now that the VM driver stages the CA host-only and runs the supervisor on the host. Inspect the driver-staged host artifact (digest-compared against the source), assert the guest overlay contains no CA material at the old mount paths, and keep the serial-console no-disclosure check. The workload validation scripts also ran curl, which exists only in the container-engine fixture image; the Kubernetes and VM lanes use the uv-based workload image and died instantly with "canonical main process already finished". Rewrite the main-process, exec, and lifecycle validation scripts in python3 (present in every lane's workload image). The network-policy binary identity is the resolved /proc/<pid>/exe target, so the versioned python3.12 binaries are allowlisted rather than the unversioned symlinks. The hostname mismatch check keeps its semantics: same reachable fixture, mismatched TLS SNI, and the failure must specifically be hostname verification. Addresses GATOR-291f43b2-04. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(kubernetes): allow admission-added labels on destination-trust ConfigMaps The destination-trust ConfigMap validation compared the whole label map for equality. A mutating admission webhook that attaches an organizational label to the created ConfigMap made the create response fail validation, and the retained ConfigMap then failed every subsequent create/start through the existing-object path, blocking sandbox launches despite the ownership labels, immutable flag, digest, and certificate bytes remaining correct. Validate the gateway-owned label pairs individually and allow unrelated labels; the exact trust-material checks are unchanged. Covered by a regression test with an admission-added label and a missing gateway-owned label. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(podman): close supervisor-refresh rollback gaps The supervisor-only trust refresh renamed the previous supervisor to a random rollback name and then performed the token-secret lookup. A transient lookup failure returned with the supervisor still renamed, where a normal delete could never find it, and the rollback rename and final rollback removal ignored their own failures, leaving orphans with attached control and channel resources. Complete every fallible preparation before the rename. On a failed replacement, restore the canonical name; if the restore itself fails, return an error naming the rollback container so the orphan stays identifiable — a later retry still succeeds because the canonical name is absent and the replacement path recreates a fresh supervisor. Treat not-found during the final rollback removal as success and name the orphan in both the log and the error on other removal failures; a retry observes the fresh canonical supervisor and resumes the normal start path. Covered by failure-injection tests for each transaction boundary: preparation failure before any rename, replacement failure restoring the canonical name, and restore failure naming the orphaned rollback container. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * docs: address destination-trust review corrections The gateway-configuration RFC documented an additional_ca_cert_paths list that the strict configuration parser never accepted; document the additional_ca_bundle single-path setting the parser actually enforces. The debug-openshell-cluster skill named the wrong generation annotation (openshell.ai/network-supervisor-trust-generation) and described digest argv and network-init canonicalization checks the implementation does not perform. Correct it to openshell.ai/additional-ca-generation and describe the actual delivery: the supervisor Pod mounts the immutable ConfigMap read-only and receives --additional-ca-bundle for the mount path, with the driver validating the digest annotation and exact bytes per creation. Classify the destination-trust generation hashes as deterministic identity derivation rather than authenticity proofs, following the openshell-crypto exclusions table: trust decisions come from the gateway's normalization, immutable staging, exact-byte validation, and TLS verification. Record the classification in the crypto README and at each hashing site. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): reconcile additional CA tests with suspend/resume and auth changes Upstream's suspend/resume lifecycle took StartSandboxRequest fields 7 and 8, so the durable-snapshot field moves from 7 to 9; update the wire-format test to assert the field-9 key (0x4a) and initialize the new resume controls. Repair the cli test module after the auth opt-in test merged alongside the environment-guard helpers, and initialize the gateway driver-startup context's network-trust fields in the shared supervision test. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(server): box start-recovery futures for clippy large-futures The durable-snapshot field grows StartSandboxRequest enough that the start-recovery match and the recovery-expiry test exceed the large-futures limit under -D warnings. Box::pin both futures, matching the pattern the recovery path used before the suspend/resume rework. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> --------- Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
openshell-driver-docker
Docker-backed compute driver for local and remote OpenShell gateways.
The driver uses bollard to manage sandbox resources through the configured
Docker API socket. When socket_path is unset, it selects the first standard
local socket that responds to an API ping. An explicitly selected Docker driver
falls back to /var/run/docker.sock when no candidate responds.
When the gateway configures [openshell.gateway.otlp], the in-process driver
exports spans to the same OTLP/gRPC collector as
openshell-driver-docker. The standalone driver accepts
OPENSHELL_OTLP_ENDPOINT, continues W3C trace context from gateway RPC
metadata, and flushes spans during graceful shutdown.
Runtime Model
Caller driver config is disabled by default. Existing volumes require administrator-controlled approval labels; raw bind mounts have no supported label resolver and are denied under enforcement. GPU devices are temporarily exempt. Inspect labels again before launch, restart, and during reconciliation. See resource admission configuration.
The driver creates two containers for each sandbox:
openshell-sandboxis PID 1 in the workload container. It owns the workload process tree, seccomp notification broker, mandatory Landlock baseline, binary identity, exec/signal/wait/PTY operations, and loopback forwarding.openshell-supervisorruns in a separate companion container. It owns the gateway session, policy engine, credentials, interception CA, SSH relay, L7 inspection, DNS policy, and external upstream connections.
Both containers are non-root, request no capabilities, and set no-new-privileges. A shared named volume carries the authenticated Unix socket and sandbox bootstrap material. A second supervisor-only volume carries the supervisor JWT and gateway client credentials and is never mounted into the workload.
The workload uses network_mode=none. Its seccomp user-notification broker
mediates every supported TCP and DNS operation, attributes it to the calling
binary, and sends the request across the private channel. The supervisor
authorizes the request before it opens an upstream connection. Docker's absent
workload network is the mandatory outer fence if mediation fails or is
bypassed. The trusted supervisor companion originates approved egress. Its
operator-owned supervisor_network_mode defaults to host. Set bridge to
run the supervisor on Docker's bridge network; auto also selects host.
The driver copies trusted runtime bytes from the configured supervisor image through the Docker archive API. No workload launch depends on a host bind mount or a tool supplied by the workload image, so the same path works with local, remote, and VM-backed Docker daemons.
Corporate Proxy Egress
[openshell.drivers.docker] accepts the operator-owned corporate proxy fields
https_proxy, no_proxy, proxy_auth_file,
proxy_auth_allow_insecure, proxy_connect_by_hostname, and
proxy_ca_bundle. Sandbox environment, image contents, and per-sandbox driver
configuration cannot select or override them.
The driver validates the proxy URL and cross-field relationships at gateway
startup. It bounded-reads proxy credentials and the PEM CA bundle before use.
Missing, unreadable, empty, oversized, malformed, or certificate-free CA files
fail closed. A CA bundle requires https_proxy, although the proxy URL may be
http:// when the proxy intercepts destination TLS.
For each supervisor launch, Docker copies the credential and CA contents into
the existing supervisor-only named volume. It passes only the fixed container
paths /.openshell/supervisor/upstream-proxy-auth and
/.openshell/supervisor/upstream-proxy-ca-bundle.pem to the supervisor. It
does not bind-mount the gateway-host files, which preserves remote-daemon
support and keeps host paths out of workload container metadata.
The supervisor trusts the configured CA for its TLS connection to an HTTPS proxy and for destination certificates re-signed by a TLS-intercepting proxy. It also includes the corporate root in the generated combined trust bundle used by workload processes.
Identity and Workspace
Before creating the workload, the driver pins the image ID and reads its
passwd/group databases through a stopped metadata container. It resolves the
admitted policy identity, or the image Config.User fallback, into one exact
non-root UID, primary GID, and supplementary-group set. Docker launches
openshell-sandbox with that identity, and the sandbox uses the same identity
for every canonical and exec process. UID or GID zero and unresolved symbolic
identities are rejected.
An absolute OCI working directory becomes the workspace. An empty, root (/),
or explicit /sandbox declaration uses /sandbox. Any other workdir must
already exist without symlink components. The resolved identity must be able to
traverse every parent and write and enter the workdir; OpenShell does not
change its ownership or mode.
Image VOLUME declarations and user mounts must not cover the workdir, one of
its parents, or the reserved /.openshell runtime/channel tree. OpenShell asks
the kernel to validate access under the final identity, so POSIX ACL and host
LSM decisions remain authoritative.
Container Contract
| Setting | Purpose |
|---|---|
Exact non-root user and group_add |
Gives sandbox and workload the same immutable UID/GID/group identity required for capability-free observation. |
cap_drop = ALL, no cap_add, no-new-privileges |
Prevents either container from acquiring Linux capabilities. |
| Docker default seccomp and AppArmor profiles | Retains runtime hardening; startup confirmation fails closed if nested seccomp notification is unavailable. |
network_mode = none on the workload |
Removes direct external routes. |
network_mode = host on the supervisor |
Lets the trusted supervisor reach the gateway's primary loopback listener and originate approved upstream connections. |
restart_policy = no |
Keeps canonical main-process exit terminal. |
PidsLimit |
Applies the configured sandbox PID budget. Omit sandbox_pids_limit to use OpenShell's default. Explicit zero is invalid. |
| Private named volumes | Separate volumes carry sandbox bootstrap keys, the runtime-initiated Unix socket, and supervisor-only credentials. The supervisor mounts bootstrap keys read-only and only the socket volume writable. |
In-memory /run/openshell-supervisor-ca tmpfs |
Holds only the public supervisor CA certificate and trust bundle without making all of /run writable. |
| CDI GPU request | Assigns the exact validated CDI devices requested by driver config or count-based selection. |
New containers dial the supervisor at /.openshell/channel/control/supervisor/control.sock.
The runtime remains the TLS/gRPC server; the supervisor verifies its pinned
identity and supplies the existing bearer credential. Saved descriptors retain
the legacy Unix listener for existing containers, including across restart.
New generations require matching supervisor and sandbox runtime binaries.
Stop, Start, and Delete
Stop terminates the supervisor companion and stops the workload container without removing it. Docker retains the workload writable layer and attached volumes. Start stages a fresh sandbox bootstrap bundle, restarts that workload, and creates a new supervisor companion. A durably stopped sandbox stays stopped across gateway restarts.
Delete force-removes both containers, the driver-owned runtime volumes, and the host-private runtime descriptor. Missing or altered descriptor and channel resources fail closed; the driver does not run an older combined-supervisor layout.
Driver Config Mounts
The gateway forwards the docker block from --driver-config-json. Supported
mount types are:
bind: an absolute daemon-host path, allowed only when[openshell.drivers.docker].enable_bind_mounts = trueand label admission is explicitly disabled.volume: an existing named volume. The driver never creates or removes a user-supplied volume. Bind-backed local volumes requireenable_bind_mounts = true.tmpfs: an in-memory filesystem with optional size and mode.
Host bind mounts are disabled by default because they expose daemon-host paths to sandbox requests. User bind and volume mounts are read-only by default. Targets must be absolute, normalized paths and cannot overlap the workspace root or OpenShell control paths.
Example:
docker volume create openshell-work
openshell sandbox create \
--driver-config-json '{"docker":{"mounts":[{"type":"volume","source":"openshell-work","target":"/sandbox/work"}]}}' \
-- claude
Runtime Image
sandbox_runtime_image contains the statically linked musl
/openshell-sandbox binary. The driver extracts that binary as bytes and
stages it into the stopped workload. supervisor_image contains the
dynamically linked glibc /openshell-supervisor binary that runs in the
supervisor companion container. Release and gateway image builds bake
matching image tags into the binary.
Gateway session and TLS
OPENSHELL_ENDPOINT and gateway authentication material are injected only into
the supervisor companion. The workload never receives the sandbox JWT, gateway
client TLS key, policy authority, or interception CA private key.
When no endpoint is configured, the supervisor connects to
127.0.0.1:<gateway-port>. Set grpc_endpoint when the gateway is not on the
Docker daemon host. A configured HTTPS server certificate must include the
endpoint host in its subject alternative names.
In host mode, the driver publishes host loopback as the backend address for
host.openshell.internal. Policy DNS resolves that reserved name through the
mediated path, so policies can reach host services without a Docker bridge,
container DNS alias, or another gateway listener.
For HTTPS endpoints, the supervisor receives only the gateway CA and uses its sandbox-scoped bearer token to authenticate RPCs. User client certificates and private keys are not delivered to either container.
Host mode on Docker Desktop requires host networking enabled in Settings.
For bridge mode or a gateway endpoint using host.docker.internal, the driver
resolves Docker's host-gateway once during initialization and caches it.
A temporary container runs --help from the immutable supervisor image with
no network, no capabilities, no privilege escalation, and a read-only root
filesystem. The driver downloads Docker's generated /etc/hosts through the
archive API, preferring a unique IPv4 address and otherwise a unique IPv6
address. Resolution has a 30-second timeout; forced cleanup has a separate
10-second timeout and runs after success or failure.
The driver uses the selected address for both host aliases and the runtime
descriptor's trust pin; workload DNS and hosts files cannot select it. Host
mode with other gateway endpoints retains its gateway-address pin. An explicit
operator-owned host_gateway_ip overrides discovery and skips the temporary
container. No supervisor changes or per-sandbox discovery are required.
When grpc_endpoint is omitted, bridge mode uses host.docker.internal on
Desktop or for a wildcard gateway listener, and the concrete bind address for
other non-loopback Linux listeners. The port is the gateway bind port; remapped
container publications require an explicit endpoint. Native Linux loopback-only
listeners cannot serve a bridge supervisor and produce an actionable error.
Explicit endpoints retain TLS verification and are never rewritten; bridge
rejects loopback/unspecified endpoint addresses. Host services bound only to
loopback are also unreachable from a native Linux bridge supervisor.
The selected network mode does not change the sandbox's mandatory Landlock and seccomp qualification.
The supervisor owns these security-critical variables:
OPENSHELL_ENDPOINTOPENSHELL_SANDBOX_IDOPENSHELL_SANDBOXOPENSHELL_SANDBOX_TOKEN_FILEOPENSHELL_SSH_SOCKET_PATHOPENSHELL_MAIN_PROCESS_SPECOPENSHELL_TLS_CAwhen HTTPS is enabled
Template and sandbox environment is encoded in the protected bootstrap and exposed only to workload children. Workload input cannot override security-critical supervisor variables.