From 2c31ac7e606c4e175288cee3656b23192a1a36b0 Mon Sep 17 00:00:00 2001 From: Jesse Jaggars Date: Tue, 27 Oct 2026 18:31:11 +0000 Subject: [PATCH] feat(network): support additional destination CAs (#3292) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(network): support additional destination CAs Signed-off-by: Jesse Jaggars * fix(network): harden additional CA propagation Signed-off-by: Jesse Jaggars * fix(network): narrow additional CA review scope Signed-off-by: Jesse Jaggars * fix(network): harden additional CA rotation Signed-off-by: Jesse Jaggars * feat(network): complete additional CA delivery Signed-off-by: Jesse Jaggars * fix(docker): remove stale runtime fixture field Signed-off-by: Jesse Jaggars * fix(network): adapt additional CA series to upstream main Resolve rebase fallout against current upstream main after the series was rebased 214 commits forward: - follow upstream #3110: supervisor→gateway TLS is CA-only, so configure_gateway_tls and its isolation tests drop the client identity - follow upstream tls_secret_names reduction to [String; 1] in the Podman driver while keeping the network trust bundle delivery - port [openshell.supervisor.network] rendering into the openshell.gatewayConfigToml serializer in _toml.tpl - adapt tests to upstream struct/function changes (DriverStartupContext, SandboxRunConfig, IsolationSpecInput, supervisor_pod, VM bootstrap image) - drop conformance-artifact-prefix plumbing removed upstream Signed-off-by: Jesse Jaggars * refactor(network): adopt upstream additional CA bundle surface Unify the additional destination CA series on the mechanism merged to main in #3996: one operator pre-assembled PEM bundle, the --additional-ca-bundle supervisor flag, and the /etc/openshell/tls/additional/ca-bundle.pem staging path. - Replace the gateway-level additional_ca_cert_paths list with a single additional_ca_bundle file validated and staged at gateway startup. - Restore the upstream supervisor argument path and fold-into-roots behavior; drop the --network-additional-ca-bundle and --network-additional-ca-digest contracts. - Keep gateway normalization bounds, content-addressed staging, digest-based driver reconciliation, and stop/start rotation; the freshness check now runs at the gateway/driver boundary instead of the supervisor boundary. - Reject additional_ca_bundle in operator driver JSON config; standalone drivers self-source the bundle from --sandbox-additional-ca-bundle with the same validation. - Extend the Kubernetes ConfigMap name digest segment to keep the full 63-byte budget after the prefix rename. - Box the persisted-sandbox reconciliation futures so callers stay under the clippy::large_futures threshold. Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and tests, e2e wiring, gateway configuration docs, and the cluster debugging skill to the single mechanism. Signed-off-by: Jesse Jaggars * fix(ci): make gateway startup tests wrap-tolerant and refresh example lockfile The network trust startup tests asserted on raw substrings of gateway diagnostics, but the CLI renderer wraps output at a fixed width with continuation markers and hard-breaks long paths. Compare assertions on whitespace- and continuation-squashed text so they hold regardless of wrap position. Format the e2e workspace (separate cargo workspace, not covered by the root cargo fmt run) and refresh examples/isolation-backend-v8/Cargo.lock for the libc dependency added to the network trust crate. Signed-off-by: Jesse Jaggars * fix(podman): reuse TLS CA secret during stopped-pair trust replacement The additional CA e2e never ran in CI before the first maintainer test dispatch, and its local container inspection was broken in two ways: it located the supervisor container through e2e network membership (the supervisor is host-networked or runtime-bridged, so the filter matched the CI job container instead), and it required the gateway mTLS CA as a bind mount (Podman delivers it as a container secret when user namespaces remap UIDs). Select the supervisor by managed labels and accept the secret delivery form. The lifecycle leg then exposed a driver bug: stopped-pair trust replacement re-created the per-sandbox TLS CA secret, colliding with the secret still owned by the renamed stopped pair. Mirror the token and resolver secret handling: require and reuse the existing secret during replacement instead of creating it. Validated end-to-end with the Podman additional CA e2e lane. Signed-off-by: Jesse Jaggars * fix(server): restore upstream recovery semantics for start-request failures Recovery no longer marks sandboxes Error/InvalidPersistedSandbox when driver_start_request fails during the startup or lifecycle sweeps. Gateway configuration problems (for example a missing launch signer) must not permanently fail unrelated sandbox records; the sweep logs a warning and skips them, matching the behavior before the additional CA work. Sandbox-record failures surface through the normal start path. Signed-off-by: Jesse Jaggars * refactor(podman): refresh stale supervisors without replacing workloads Replace the stopped-pair trust reconciliation with supervisor-only recreation. The destination-trust generation label moves from the workload to the supervisor container, and start_sandbox recreates only the supervisor when its generation is missing or stale. The workload container, its volumes, its secrets, and its workspace data are never replaced: the additional CA bundle is mounted only into the supervisor. Running sandboxes are never gated on trust labels, and recovery without fresh launch credentials keeps the existing supervisor. The intentional -removal fence machinery in the watcher existed only for workload replacement and is removed with it. Signed-off-by: Jesse Jaggars * refactor(docker): drop workload replacement for destination-trust changes Delete the stopped-workload trust reconciliation path and all of its machinery: the trust-generation label on workload containers, the generation match check in the start fence, the stopped-sandbox replacement, the bounded workspace archive copy with its 512 MiB cap and regular-file-only validation, and the boundary-state rollback helpers. No replacement machinery is needed: the trusted supervisor container is already re-created from the gateway's current configuration on every start via ensure_control_process_for_container, so stop/start picks up the current additional CA bundle while the workload container, its volumes, and its workspace data are never replaced. Signed-off-by: Jesse Jaggars * docs(configuration): drop reconciliation wording from additional CA lifecycle The stop/start lifecycle description already matches the supervisor-only refresh model; remove the two remaining references to restart reconciliation of workload containers. Signed-off-by: Jesse Jaggars * fix(network): port additional CA test fixtures to the crypto interface Route the gateway TLS isolation test PKI, the destination CA source fixtures, and the gateway startup trust tests through openshell_crypto::pki now that rcgen access is centralized there: generate_keypair/self_signed/signed_by replace the direct rcgen calls and CertifiedKey field access follows the wrapper's shape. Signed-off-by: Jesse Jaggars * fix(e2e): repair additional CA lane for Docker and non-container builds The first maintainer test dispatch revealed two latent issues in the additional CA e2e that no local run had exercised: the Podman gateway CA secret helper referenced ContainerEngine without the container feature gate, breaking Kubernetes and VM builds of the test, and the Docker branch asserted a client ca.crt bind mount that the Docker driver never creates. Gate the helper like its callers, and assert Docker's actual delivery path instead: the gateway mTLS CA staged as tls/ca.pem inside the read-only supervisor state volume, selected through the OPENSHELL_TLS_CA environment pointer. Signed-off-by: Jesse Jaggars * fix(helm): restore alphabetical README ordering for supervisor keys The helm:docs check regenerates the chart README with helm-docs, which orders rows alphabetically; a conflict resolution had placed supervisor.network.additionalCaConfigMapName after the supervisor.podSecurityContext rows. Regenerate with mise run helm:docs. Signed-off-by: Jesse Jaggars * chore: refresh testkit lockfile for openshell-core libc dependency Signed-off-by: Jesse Jaggars * fix(driver-podman): stage operator CA bundle as driver-owned artifact Standalone Podman drivers accepting --sandbox-additional-ca-bundle previously kept the operator's source path as the trust artifact path. The immutable-artifact verification performed on every create/start requires exactly mode 0444, which an operator-owned source path cannot guarantee, so a readable-but-writable source failed reconciliation. Read the operator bundle with the same bounded, no-symlink validation, then stage the normalized material into a driver-owned artifact inside a private tempdir held for the driver process lifetime. The artifact is created 0600, synced, and only then restricted to 0444 so no observer ever sees a writable artifact. The operator source and its permissions are never referenced after startup. Addresses GATOR-291f43b2-02. Signed-off-by: Jesse Jaggars * fix(driver-podman): pin retained workload image during supervisor refresh Supervisor-only trust reconciliation re-entered create_sandbox_with_existing_token, which re-pulled and re-resolved the workload image tag even though the workload container is retained. If the tag moved to a different image between stop and start, identity resolution and child-environment construction would reflect the new image while the retained workload still runs the original one, leaving mode-0600 bootstrap files the retained user cannot read. During a refresh with a retained workload, inspect the stopped container and pin to the immutable image ID it was created from (Config.Image); never pull or resolve the request tag. The requested reference is still used for child metadata so the retained workload's environment is byte-identical to its original launch. Addresses GATOR-291f43b2-01. Signed-off-by: Jesse Jaggars * test(e2e): verify host-only VM trust delivery and drop curl dependency The additional-CA lane's VM assertions read the destination CA out of the guest overlay and a guest-side supervisor-argv marker — artifacts from the guest-supervisor design that no longer exist now that the VM driver stages the CA host-only and runs the supervisor on the host. Inspect the driver-staged host artifact (digest-compared against the source), assert the guest overlay contains no CA material at the old mount paths, and keep the serial-console no-disclosure check. The workload validation scripts also ran curl, which exists only in the container-engine fixture image; the Kubernetes and VM lanes use the uv-based workload image and died instantly with "canonical main process already finished". Rewrite the main-process, exec, and lifecycle validation scripts in python3 (present in every lane's workload image). The network-policy binary identity is the resolved /proc//exe target, so the versioned python3.12 binaries are allowlisted rather than the unversioned symlinks. The hostname mismatch check keeps its semantics: same reachable fixture, mismatched TLS SNI, and the failure must specifically be hostname verification. Addresses GATOR-291f43b2-04. Signed-off-by: Jesse Jaggars * fix(kubernetes): allow admission-added labels on destination-trust ConfigMaps The destination-trust ConfigMap validation compared the whole label map for equality. A mutating admission webhook that attaches an organizational label to the created ConfigMap made the create response fail validation, and the retained ConfigMap then failed every subsequent create/start through the existing-object path, blocking sandbox launches despite the ownership labels, immutable flag, digest, and certificate bytes remaining correct. Validate the gateway-owned label pairs individually and allow unrelated labels; the exact trust-material checks are unchanged. Covered by a regression test with an admission-added label and a missing gateway-owned label. Signed-off-by: Jesse Jaggars * fix(podman): close supervisor-refresh rollback gaps The supervisor-only trust refresh renamed the previous supervisor to a random rollback name and then performed the token-secret lookup. A transient lookup failure returned with the supervisor still renamed, where a normal delete could never find it, and the rollback rename and final rollback removal ignored their own failures, leaving orphans with attached control and channel resources. Complete every fallible preparation before the rename. On a failed replacement, restore the canonical name; if the restore itself fails, return an error naming the rollback container so the orphan stays identifiable — a later retry still succeeds because the canonical name is absent and the replacement path recreates a fresh supervisor. Treat not-found during the final rollback removal as success and name the orphan in both the log and the error on other removal failures; a retry observes the fresh canonical supervisor and resumes the normal start path. Covered by failure-injection tests for each transaction boundary: preparation failure before any rename, replacement failure restoring the canonical name, and restore failure naming the orphaned rollback container. Signed-off-by: Jesse Jaggars * docs: address destination-trust review corrections The gateway-configuration RFC documented an additional_ca_cert_paths list that the strict configuration parser never accepted; document the additional_ca_bundle single-path setting the parser actually enforces. The debug-openshell-cluster skill named the wrong generation annotation (openshell.ai/network-supervisor-trust-generation) and described digest argv and network-init canonicalization checks the implementation does not perform. Correct it to openshell.ai/additional-ca-generation and describe the actual delivery: the supervisor Pod mounts the immutable ConfigMap read-only and receives --additional-ca-bundle for the mount path, with the driver validating the digest annotation and exact bytes per creation. Classify the destination-trust generation hashes as deterministic identity derivation rather than authenticity proofs, following the openshell-crypto exclusions table: trust decisions come from the gateway's normalization, immutable staging, exact-byte validation, and TLS verification. Record the classification in the crypto README and at each hashing site. Signed-off-by: Jesse Jaggars * fix(network): reconcile additional CA tests with suspend/resume and auth changes Upstream's suspend/resume lifecycle took StartSandboxRequest fields 7 and 8, so the durable-snapshot field moves from 7 to 9; update the wire-format test to assert the field-9 key (0x4a) and initialize the new resume controls. Repair the cli test module after the auth opt-in test merged alongside the environment-guard helpers, and initialize the gateway driver-startup context's network-trust fields in the shared supervision test. Signed-off-by: Jesse Jaggars * fix(server): box start-recovery futures for clippy large-futures The durable-snapshot field grows StartSandboxRequest enough that the start-recovery match and the recovery-expiry test exceed the large-futures limit under -D warnings. Box::pin both futures, matching the pattern the recovery path used before the suspend/resume rework. Signed-off-by: Jesse Jaggars --------- Signed-off-by: Jesse Jaggars --- .github/workflows/branch-e2e.yml | 38 +- .github/workflows/e2e-docker-test.yml | 2 +- .github/workflows/e2e-vm-test.yml | 5 + Cargo.lock | 7 + crates/openshell-core/Cargo.toml | 4 +- crates/openshell-core/src/driver_utils.rs | 2 +- crates/openshell-core/src/grpc_client.rs | 164 ++- crates/openshell-core/src/lib.rs | 2 + crates/openshell-core/src/network_trust.rs | 427 ++++++ crates/openshell-crypto/README.md | 1 + crates/openshell-driver-docker/src/lib.rs | 175 ++- crates/openshell-driver-docker/src/main.rs | 2 +- crates/openshell-driver-docker/src/tests.rs | 138 +- crates/openshell-driver-kubernetes/Cargo.toml | 1 + .../openshell-driver-kubernetes/src/driver.rs | 459 +++++- .../src/lifecycle_tests.rs | 1 + .../openshell-driver-kubernetes/src/main.rs | 1 + .../src/sandbox_runtime.rs | 133 +- crates/openshell-driver-podman/Cargo.toml | 3 +- crates/openshell-driver-podman/src/client.rs | 118 +- crates/openshell-driver-podman/src/config.rs | 18 + .../openshell-driver-podman/src/container.rs | 380 ++--- crates/openshell-driver-podman/src/driver.rs | 898 +++++++++++- crates/openshell-driver-podman/src/grpc.rs | 37 +- crates/openshell-driver-podman/src/main.rs | 99 +- crates/openshell-driver-podman/src/watcher.rs | 29 +- crates/openshell-driver-vm/Cargo.toml | 1 + crates/openshell-driver-vm/src/driver.rs | 736 +++++++++- crates/openshell-driver-vm/src/main.rs | 59 + crates/openshell-gateway/Cargo.toml | 2 + crates/openshell-gateway/src/lib.rs | 54 +- crates/openshell-gateway/src/vm.rs | 67 +- .../tests/network_trust_startup.rs | 238 ++++ crates/openshell-sandbox/src/child_env.rs | 46 + crates/openshell-sandbox/src/delegated.rs | 5 + crates/openshell-sandbox/src/process.rs | 58 +- crates/openshell-server/src/cli.rs | 253 ++++ .../src/compute/driver_config.rs | 12 +- crates/openshell-server/src/compute/mod.rs | 113 +- crates/openshell-server/src/config_file.rs | 88 ++ crates/openshell-server/src/lib.rs | 183 ++- crates/openshell-server/src/network_trust.rs | 1147 +++++++++++++++ crates/openshell-server/src/test_support.rs | 9 + deploy/helm/openshell/README.md | 72 + deploy/helm/openshell/README.md.gotmpl | 71 + .../openshell/templates/_gateway-workload.tpl | 27 +- deploy/helm/openshell/templates/_toml.tpl | 10 + .../helm/openshell/templates/clusterrole.yaml | 18 + deploy/helm/openshell/templates/role.yaml | 14 +- .../helm/openshell/templates/rolebinding.yaml | 4 +- .../openshell/tests/gateway_config_test.yaml | 29 + .../tests/network_additional_ca_test.yaml | 223 +++ deploy/helm/openshell/values.yaml | 3 + docs/how-it-works/gateways/configuration.mdx | 113 +- e2e/rust/Cargo.toml | 6 + e2e/rust/e2e-docker.sh | 5 + e2e/rust/e2e-kubernetes.sh | 8 + e2e/rust/e2e-podman.sh | 5 + e2e/rust/e2e-vm.sh | 58 + e2e/rust/tests/additional_ca.rs | 1232 +++++++++++++++++ e2e/support/gateway-common.sh | 92 ++ e2e/with-docker-gateway.sh | 37 + e2e/with-kube-gateway.sh | 53 + e2e/with-podman-gateway.sh | 46 + examples/governance-interceptor/Cargo.lock | 1 + examples/isolation-backend-v8/Cargo.lock | 1 + .../Cargo.lock | 1 + proto/compute_driver.proto | 6 + rfc/0003-gateway-configuration/README.md | 11 +- skills/debug-openshell-cluster/SKILL.md | 96 +- tests/testkit/Cargo.lock | 1 + 71 files changed, 8024 insertions(+), 434 deletions(-) create mode 100644 crates/openshell-core/src/network_trust.rs create mode 100644 crates/openshell-gateway/tests/network_trust_startup.rs create mode 100644 crates/openshell-server/src/network_trust.rs create mode 100644 deploy/helm/openshell/tests/network_additional_ca_test.yaml create mode 100644 e2e/rust/tests/additional_ca.rs diff --git a/.github/workflows/branch-e2e.yml b/.github/workflows/branch-e2e.yml index db4c96b93..8d2354a11 100644 --- a/.github/workflows/branch-e2e.yml +++ b/.github/workflows/branch-e2e.yml @@ -328,6 +328,19 @@ jobs: suite-matrix: >- [{"suite":"ci","runner":"ubuntu-26.04","podman_major":"5","podman_package_version":"5.7.0+ds2-3build1","conmon_package_version":"2.1.13+ds1-2","cmd":"mise run --no-deps --skip-deps e2e:podman:ci"}] + podman-additional-ca-e2e: + needs: [pr_metadata, build-binaries, build-images] + if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' + permissions: + actions: read + contents: read + packages: read + uses: ./.github/workflows/e2e-podman-test.yml + with: + image-tag: ${{ github.sha }} + suite-matrix: >- + [{"suite":"additional-ca","runner":"ubuntu-26.04","podman_major":"5","podman_package_version":"5.7.0+ds2-3build1","conmon_package_version":"2.1.13+ds1-2","cmd":"OPENSHELL_E2E_PODMAN_TEST=additional_ca mise run --no-deps --skip-deps e2e:podman"}] + vm-e2e: needs: [pr_metadata, build-binaries, build-vm-driver] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' @@ -337,6 +350,18 @@ jobs: packages: read uses: ./.github/workflows/e2e-vm-test.yml + vm-additional-ca-e2e: + needs: [pr_metadata, build-binaries, build-vm-driver] + if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' + permissions: + actions: read + contents: read + packages: read + uses: ./.github/workflows/e2e-vm-test.yml + with: + suite-name: additional-ca + test-name: additional_ca + docker-external-driver-e2e: needs: [pr_metadata, build-binaries, build-gateway-plain, build-external-drivers, build-images] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' @@ -401,8 +426,16 @@ jobs: include: - agent_sandbox_api: v1beta1 agent_sandbox_version: v1.0.3 + job_name: Kubernetes E2E (Rust smoke, Agent Sandbox v1beta1) + test_name: "" - agent_sandbox_api: v1alpha1 agent_sandbox_version: v0.4.6 + job_name: Kubernetes E2E (Rust smoke, Agent Sandbox v1alpha1) + test_name: "" + - agent_sandbox_api: v1beta1 + agent_sandbox_version: v0.5.0 + job_name: Kubernetes E2E (additional destination CA, Agent Sandbox v1beta1) + test_name: additional_ca permissions: actions: read contents: read @@ -410,8 +443,9 @@ jobs: uses: ./.github/workflows/e2e-kubernetes-test.yml with: image-tag: ${{ github.sha }} - job-name: Kubernetes E2E (Rust smoke, Agent Sandbox ${{ matrix.agent_sandbox_api }}) + job-name: ${{ matrix.job_name }} agent-sandbox-version: ${{ matrix.agent_sandbox_version }} + test-name: ${{ matrix.test_name }} kubernetes-workspace-managed-e2e: needs: [pr_metadata, build-binaries, build-images] @@ -494,7 +528,9 @@ jobs: - driver-specific-integration - docker-e2e - podman-e2e + - podman-additional-ca-e2e - vm-e2e + - vm-additional-ca-e2e - docker-external-driver-e2e - podman-external-driver-e2e - vm-external-driver-e2e diff --git a/.github/workflows/e2e-docker-test.yml b/.github/workflows/e2e-docker-test.yml index 96353688b..501ded8d3 100644 --- a/.github/workflows/e2e-docker-test.yml +++ b/.github/workflows/e2e-docker-test.yml @@ -29,7 +29,7 @@ on: required: false type: string default: >- - [{"suite":"python","cmd":"mise run --no-deps --skip-deps e2e:python","apt_packages":"","python_proto":true,"mcp":false},{"suite":"oidc-python","cmd":"mise run --no-deps --skip-deps e2e:oidc-python:docker","apt_packages":"","python_proto":true,"mcp":false},{"suite":"oidc-pkce-docker","cmd":"mise run --no-deps --skip-deps e2e:oidc-pkce:docker","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"rust-docker","cmd":"mise run --no-deps --skip-deps e2e:rust","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"rust-docker-push","cmd":"mise run --no-deps --skip-deps e2e:rust:push","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"oci-image","cmd":"mise run --no-deps --skip-deps e2e:oci-image","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"mcp","cmd":"mise run --no-deps --skip-deps e2e:mcp","apt_packages":"","python_proto":false,"mcp":true}] + [{"suite":"python","cmd":"mise run --no-deps --skip-deps e2e:python","apt_packages":"","python_proto":true,"mcp":false},{"suite":"oidc-python","cmd":"mise run --no-deps --skip-deps e2e:oidc-python:docker","apt_packages":"","python_proto":true,"mcp":false},{"suite":"oidc-pkce-docker","cmd":"mise run --no-deps --skip-deps e2e:oidc-pkce:docker","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"rust-docker","cmd":"mise run --no-deps --skip-deps e2e:rust","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"rust-docker-push","cmd":"mise run --no-deps --skip-deps e2e:rust:push","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"oci-image","cmd":"mise run --no-deps --skip-deps e2e:oci-image","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"additional-ca","cmd":"OPENSHELL_E2E_DOCKER_TEST=additional_ca mise run --no-deps --skip-deps e2e:docker","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"mcp","cmd":"mise run --no-deps --skip-deps e2e:mcp","apt_packages":"","python_proto":false,"mcp":true}] permissions: actions: read diff --git a/.github/workflows/e2e-vm-test.yml b/.github/workflows/e2e-vm-test.yml index 57e04c32b..28b4c81db 100644 --- a/.github/workflows/e2e-vm-test.yml +++ b/.github/workflows/e2e-vm-test.yml @@ -22,6 +22,10 @@ on: required: false type: string default: e2e:vm + test-name: + required: false + type: string + default: "" permissions: actions: read @@ -55,4 +59,5 @@ jobs: - name: Run tests env: E2E_TASK: ${{ inputs.e2e-task }} + OPENSHELL_E2E_VM_TEST: ${{ inputs.test-name }} run: mise run --no-deps --skip-deps "$E2E_TASK" diff --git a/Cargo.lock b/Cargo.lock index 9c8d0974d..f8675b9c6 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4372,8 +4372,10 @@ dependencies = [ "base64 0.22.1", "chrono", "glob", + "h2", "ipnet", "jsonwebtoken", + "libc", "miette", "nix 0.29.0", "openshell-crypto", @@ -4514,6 +4516,7 @@ dependencies = [ "rustls", "serde", "serde_json", + "sha2 0.10.9", "temp-env", "thiserror 2.0.20", "tokio", @@ -4595,6 +4598,7 @@ dependencies = [ "rustix 1.1.4", "serde", "serde_json", + "sha2 0.10.9", "tar", "temp-env", "tempfile", @@ -4665,6 +4669,7 @@ dependencies = [ "polling", "prost", "prost-types", + "rcgen", "rustix 1.1.4", "serde", "serde_json", @@ -4710,6 +4715,7 @@ dependencies = [ "miette", "nix 0.29.0", "openshell-core", + "openshell-crypto", "openshell-driver-docker", "openshell-driver-kubernetes", "openshell-driver-mxc", @@ -4717,6 +4723,7 @@ dependencies = [ "openshell-otel", "openshell-policy", "openshell-server", + "rcgen", "rustix 1.1.4", "serde", "serde_json", diff --git a/crates/openshell-core/Cargo.toml b/crates/openshell-core/Cargo.toml index d03aafee9..137744363 100644 --- a/crates/openshell-core/Cargo.toml +++ b/crates/openshell-core/Cargo.toml @@ -46,6 +46,7 @@ openshell-otel = { path = "../openshell-otel", optional = true } [target.'cfg(unix)'.dependencies] nix = { workspace = true } rustix = { workspace = true } +libc = "0.2" [features] default = ["telemetry"] @@ -66,9 +67,10 @@ tonic-prost-build = { workspace = true } protoc-bin-vendored = { workspace = true } [dev-dependencies] +h2 = "0.4" +rcgen = { workspace = true } tempfile = "3" tracing-subscriber = { workspace = true } -rcgen = { workspace = true } tokio-rustls = { workspace = true } [lints] diff --git a/crates/openshell-core/src/driver_utils.rs b/crates/openshell-core/src/driver_utils.rs index bb2261708..32c1ead05 100644 --- a/crates/openshell-core/src/driver_utils.rs +++ b/crates/openshell-core/src/driver_utils.rs @@ -545,7 +545,7 @@ fn read_regular_file_bounded(path: &str, max_bytes: u64) -> Result, + server_name: Option<&str>, +) -> Result { + let mut tls_config = ClientTlsConfig::new().ca_certificate(Certificate::from_pem(ca_pem)); + if let Some(server_name) = server_name.filter(|name| !name.is_empty()) { + tls_config = tls_config.domain_name(server_name); + } + ep.tls_config(tls_config) + .into_diagnostic() + .wrap_err("failed to configure TLS") +} + /// Build the plain (un-intercepted) gRPC channel. /// /// When the endpoint uses `https://`, server-authenticated TLS is configured @@ -405,27 +423,12 @@ async fn build_plain_channel(endpoint: &str) -> Result { .into_diagnostic() .wrap_err_with(|| format!("failed to read CA cert from {ca_path}"))?; - // Trust only the configured CA — this is the chart's internal CA - // that signs the gateway's internal server certificate. The gateway uses SNI-based - // certificate selection to present this internal cert to supervisor - // connections, so no public root trust is needed here. - // - // Do NOT add `.with_native_roots()` or `.with_webpki_roots()` here: - // the supervisor runs inside the user-selected sandbox image - // (Docker/Podman drivers), and broadening the trust store would let - // an attacker who controls the image + DNS present a publicly valid - // certificate and intercept the supervisor→gateway TLS connection. - let mut tls_config = ClientTlsConfig::new().ca_certificate(Certificate::from_pem(ca_pem)); - if let Ok(server_name) = std::env::var(sandbox_env::GATEWAY_TLS_SERVER_NAME) - && !server_name.is_empty() - { - tls_config = tls_config.domain_name(server_name); - } - - ep = ep - .tls_config(tls_config) - .into_diagnostic() - .wrap_err("failed to configure TLS")?; + // The gateway uses SNI-based certificate selection to present its + // internal cert to supervisor connections. Do not add native, + // webpki, destination, or child-process roots here: an image that can + // influence DNS must not broaden supervisor→gateway authentication. + let server_name = std::env::var(sandbox_env::GATEWAY_TLS_SERVER_NAME).ok(); + ep = configure_gateway_tls(ep, ca_pem, server_name.as_deref())?; } ep.connect() @@ -462,6 +465,125 @@ impl Drop for ClientSpanStatus { } } +#[cfg(test)] +mod gateway_tls_isolation_tests { + use super::*; + use rcgen::{CertificateParams, IsCa}; + use rustls::pki_types::{CertificateDer, PrivateKeyDer}; + use std::io::Cursor; + use tokio::net::TcpListener; + use tokio_rustls::TlsAcceptor; + + #[allow(clippy::struct_field_names)] + struct TestPki { + ca_pem: Vec, + server_cert_pem: Vec, + server_key_pem: Vec, + } + + fn test_pki(server_name: &str) -> TestPki { + let mut ca_params = CertificateParams::new(Vec::::new()).unwrap(); + ca_params.is_ca = IsCa::Ca(rcgen::BasicConstraints::Unconstrained); + ca_params.key_usages = vec![ + rcgen::KeyUsagePurpose::KeyCertSign, + rcgen::KeyUsagePurpose::CrlSign, + ]; + let ca_key = openshell_crypto::pki::generate_keypair().unwrap(); + let ca = openshell_crypto::pki::self_signed(ca_params, &ca_key).unwrap(); + + let server_key = openshell_crypto::pki::generate_keypair().unwrap(); + let server_params = CertificateParams::new(vec![server_name.to_string()]).unwrap(); + let server = + openshell_crypto::pki::signed_by(server_params, &server_key, &ca, &ca_key).unwrap(); + TestPki { + ca_pem: ca.pem().into_bytes(), + server_cert_pem: server.pem().into_bytes(), + server_key_pem: server_key.serialize_pem().unwrap().into_bytes(), + } + } + + async fn start_h2_tls_server(pki: &TestPki) -> std::net::SocketAddr { + let certificates = rustls_pemfile::certs(&mut Cursor::new(&pki.server_cert_pem)) + .collect::>, _>>() + .unwrap(); + let key: PrivateKeyDer<'static> = + rustls_pemfile::private_key(&mut Cursor::new(&pki.server_key_pem)) + .unwrap() + .unwrap(); + let mut config = rustls::ServerConfig::builder() + .with_no_client_auth() + .with_single_cert(certificates, key) + .unwrap(); + config.alpn_protocols = vec![b"h2".to_vec()]; + let acceptor = TlsAcceptor::from(Arc::new(config)); + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + tokio::spawn(async move { + let Ok((stream, _)) = listener.accept().await else { + return; + }; + let Ok(stream) = acceptor.accept(stream).await else { + return; + }; + if let Ok(mut connection) = h2::server::handshake(stream).await { + while let Some(request) = connection.accept().await { + if request.is_err() { + break; + } + } + } + }); + address + } + + async fn connect_with_gateway_ca( + address: std::net::SocketAddr, + pki: &TestPki, + server_name: &str, + ) -> Result { + let endpoint = Endpoint::from_shared(format!("https://{address}")).into_diagnostic()?; + configure_gateway_tls(endpoint, pki.ca_pem.clone(), Some(server_name))? + .connect() + .await + .into_diagnostic() + } + + #[tokio::test] + async fn destination_ca_cannot_authenticate_gateway() { + const GATEWAY_NAME: &str = "gateway.internal.test"; + let gateway = test_pki(GATEWAY_NAME); + let destination = test_pki(GATEWAY_NAME); + + let real_address = start_h2_tls_server(&gateway).await; + tokio::time::timeout( + Duration::from_secs(5), + connect_with_gateway_ca(real_address, &gateway, GATEWAY_NAME), + ) + .await + .expect("real gateway connection timed out") + .expect("configured gateway CA should authenticate the gateway"); + + let fake_address = start_h2_tls_server(&destination).await; + let error = tokio::time::timeout( + Duration::from_secs(5), + connect_with_gateway_ca(fake_address, &gateway, GATEWAY_NAME), + ) + .await + .expect("fake gateway rejection timed out") + .expect_err("destination-signed fake gateway must be rejected"); + assert!(format!("{error:?}").contains("certificate"), "{error:?}"); + + let mismatch_address = start_h2_tls_server(&gateway).await; + tokio::time::timeout( + Duration::from_secs(5), + connect_with_gateway_ca(mismatch_address, &gateway, "wrong.internal.test"), + ) + .await + .expect("hostname mismatch rejection timed out") + .expect_err("normal server-name verification must remain enabled"); + } +} + /// Build a Bearer-authenticated channel to the gateway with one sandbox /// instance's credentials. The first connection starts that instance's /// renewal task. diff --git a/crates/openshell-core/src/lib.rs b/crates/openshell-core/src/lib.rs index d0db82191..0588a9974 100644 --- a/crates/openshell-core/src/lib.rs +++ b/crates/openshell-core/src/lib.rs @@ -40,6 +40,7 @@ pub mod mcp; pub mod metadata; pub mod middleware; pub mod net; +pub mod network_trust; #[cfg(feature = "oauth")] pub mod oauth; pub mod paths; @@ -79,6 +80,7 @@ pub use error::{ComputeDriverError, Error, Result}; pub use metadata::{ GetResourceVersion, ObjectId, ObjectLabels, ObjectName, ObjectWorkspace, SetResourceVersion, }; +pub use network_trust::NetworkSupervisorTrustBundle; pub use sandbox_session::{SandboxSessionId, SandboxSessionIdError}; /// Build version string derived from git metadata. diff --git a/crates/openshell-core/src/network_trust.rs b/crates/openshell-core/src/network_trust.rs new file mode 100644 index 000000000..cd0a349eb --- /dev/null +++ b/crates/openshell-core/src/network_trust.rs @@ -0,0 +1,427 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Shared representation of gateway-configured destination trust material. +//! +//! The gateway owns validation and creation of this material. This type keeps +//! the normalized certificate bytes separate from gateway mTLS material while +//! giving compute-driver adapters read-only access to the data they must stage +//! for the network supervisor. + +use std::fmt; +use std::path::{Path, PathBuf}; +use std::sync::Arc; + +/// Maximum number of normalized PEM bytes accepted for destination trust. +/// +/// This bound is shared by the gateway normalization boundary and all compute +/// driver consumers. Kubernetes counts the `ca.crt` key and value toward its +/// 1 MiB `ConfigMap` data limit, so reserve the six key bytes here to keep every +/// supported driver on the same deployable boundary. +pub const MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES: usize = 1024 * 1024 - "ca.crt".len(); + +/// Protected resource metadata key recording the destination-trust generation +/// used to build a sandbox startup artifact. +/// +/// Kubernetes stores this value in an annotation because a `sha256:` +/// digest exceeds the 63-character label-value limit. Local container drivers +/// may store the same protected key in a container label. +pub const NETWORK_SUPERVISOR_TRUST_GENERATION_KEY: &str = "openshell.ai/additional-ca-generation"; + +/// Explicit generation marker for an omitted destination-trust configuration. +/// +/// An explicit value, rather than an absent marker, lets a stopped sandbox +/// distinguish a removal from a resource created by an older gateway. +pub const NETWORK_SUPERVISOR_TRUST_GENERATION_NONE: &str = "none"; + +/// Normalized, gateway-owned trust material for sandbox destination TLS. +/// +/// The PEM bytes contain only canonical X.509 certificate blocks. The type is +/// deliberately distinct from gateway client TLS material: its contents must +/// never be used to authenticate the gateway control plane. +#[derive(Clone, Eq, PartialEq)] +pub struct NetworkSupervisorTrustBundle { + normalized_pem: Arc<[u8]>, + certificate_count: usize, + digest: String, + artifact_path: PathBuf, +} + +impl NetworkSupervisorTrustBundle { + /// Construct a bundle from already-normalized certificate material. + /// + /// Callers are responsible for validating and canonicalizing the PEM + /// before constructing the shared value. The gateway performs that work + /// at startup; drivers only consume the resulting read-only data. + #[must_use] + pub fn new( + normalized_pem: Vec, + certificate_count: usize, + digest: impl Into, + artifact_path: PathBuf, + ) -> Self { + Self { + normalized_pem: Arc::from(normalized_pem), + certificate_count, + digest: digest.into(), + artifact_path, + } + } + + /// Return the canonical certificate PEM bytes. + #[must_use] + pub fn pem(&self) -> &[u8] { + &self.normalized_pem + } + + /// Alias documenting that the returned bytes are normalized material. + #[must_use] + pub fn normalized_pem(&self) -> &[u8] { + self.pem() + } + + /// Return the number of certificates in the normalized bundle. + #[must_use] + pub const fn certificate_count(&self) -> usize { + self.certificate_count + } + + /// Return the non-secret digest of the normalized bundle. + #[must_use] + pub fn digest(&self) -> &str { + &self.digest + } + + /// Return the gateway-owned host path that local/VM adapters may stage. + #[must_use] + pub fn artifact_path(&self) -> &Path { + &self.artifact_path + } + + /// Return whether the bundle contains no normalized bytes. + #[must_use] + pub fn is_empty(&self) -> bool { + self.normalized_pem.is_empty() + } + + /// Verify that the staged artifact still exactly matches this startup + /// snapshot. + /// + /// Local drivers call this before building or replacing a stopped sandbox. + /// The check is deliberately bounded and follows no symlinks on Unix so a + /// changed, missing, or non-regular artifact fails closed before a runtime + /// can consume it. Diagnostics identify only the artifact path and digest; + /// certificate contents are never included. + pub fn verify_artifact(&self) -> Result<(), NetworkSupervisorTrustArtifactError> { + use std::fs::{self, OpenOptions}; + use std::io::Read as _; + + let path = self.artifact_path(); + let metadata = fs::symlink_metadata(path).map_err(|error| { + NetworkSupervisorTrustArtifactError::Read { + path: path.to_path_buf(), + error, + } + })?; + if !metadata.file_type().is_file() { + return Err(NetworkSupervisorTrustArtifactError::NotRegularFile { + path: path.to_path_buf(), + }); + } + if !has_expected_artifact_permissions(&metadata) { + return Err(NetworkSupervisorTrustArtifactError::InsecurePermissions { + path: path.to_path_buf(), + }); + } + if metadata.len() > MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES as u64 { + return Err(NetworkSupervisorTrustArtifactError::TooLarge { + path: path.to_path_buf(), + limit: MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES, + }); + } + + let mut options = OpenOptions::new(); + options.read(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.custom_flags(libc::O_CLOEXEC | libc::O_NOFOLLOW | libc::O_NONBLOCK); + } + let mut file = + options + .open(path) + .map_err(|error| NetworkSupervisorTrustArtifactError::Read { + path: path.to_path_buf(), + error, + })?; + let opened_metadata = + file.metadata() + .map_err(|error| NetworkSupervisorTrustArtifactError::Read { + path: path.to_path_buf(), + error, + })?; + if !opened_metadata.file_type().is_file() { + return Err(NetworkSupervisorTrustArtifactError::NotRegularFile { + path: path.to_path_buf(), + }); + } + if !has_expected_artifact_permissions(&opened_metadata) { + return Err(NetworkSupervisorTrustArtifactError::InsecurePermissions { + path: path.to_path_buf(), + }); + } + if opened_metadata.len() > MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES as u64 { + return Err(NetworkSupervisorTrustArtifactError::TooLarge { + path: path.to_path_buf(), + limit: MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES, + }); + } + + let capacity = usize::try_from(opened_metadata.len()).map_err(|_| { + NetworkSupervisorTrustArtifactError::TooLarge { + path: path.to_path_buf(), + limit: MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES, + } + })?; + let mut actual = Vec::with_capacity(capacity); + file.by_ref() + .take((MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES + 1) as u64) + .read_to_end(&mut actual) + .map_err(|error| NetworkSupervisorTrustArtifactError::Read { + path: path.to_path_buf(), + error, + })?; + if actual.len() > MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES { + return Err(NetworkSupervisorTrustArtifactError::TooLarge { + path: path.to_path_buf(), + limit: MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES, + }); + } + if actual != self.normalized_pem() { + return Err(NetworkSupervisorTrustArtifactError::GenerationMismatch { + path: path.to_path_buf(), + digest: self.digest.clone(), + }); + } + Ok(()) + } +} + +/// Return whether metadata has the read-only artifact permissions created by +/// the gateway. The containing `0700` directory limits access to its owner; +/// `0444` allows both root Docker and rootless Podman consumers to read a bind +/// mounted generation without allowing any process to modify it. +fn has_expected_artifact_permissions(metadata: &std::fs::Metadata) -> bool { + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + metadata.permissions().mode() & 0o7777 == 0o444 + } + #[cfg(not(unix))] + { + let _ = metadata; + true + } +} + +/// Non-secret failures while re-reading a gateway-owned staged trust artifact. +#[derive(Debug, thiserror::Error)] +pub enum NetworkSupervisorTrustArtifactError { + #[error("network additional CA artifact '{path}' could not be read: {error}")] + Read { + path: PathBuf, + #[source] + error: std::io::Error, + }, + #[error("network additional CA artifact '{path}' is not a regular file")] + NotRegularFile { path: PathBuf }, + #[error( + "network additional CA artifact '{path}' does not have the required read-only permissions" + )] + InsecurePermissions { path: PathBuf }, + #[error("network additional CA artifact '{path}' exceeds the shared {limit}-byte limit")] + TooLarge { path: PathBuf, limit: usize }, + #[error( + "network additional CA artifact '{path}' does not match the gateway startup trust generation {digest}" + )] + GenerationMismatch { path: PathBuf, digest: String }, +} + +impl fmt::Debug for NetworkSupervisorTrustBundle { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("NetworkSupervisorTrustBundle") + .field("certificate_count", &self.certificate_count) + .field("digest", &self.digest) + .field("artifact_path", &self.artifact_path) + .finish_non_exhaustive() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn debug_representation_redacts_pem_bytes() { + let pem = b"-----BEGIN CERTIFICATE-----\nnot-secret-in-a-real-fixture\n-----END CERTIFICATE-----\n"; + let bundle = NetworkSupervisorTrustBundle::new( + pem.to_vec(), + 1, + "sha256:test-digest", + PathBuf::from("/var/lib/openshell/network/additional-ca.crt"), + ); + + let debug = format!("{bundle:?}"); + assert!(debug.contains("certificate_count: 1")); + assert!(debug.contains("sha256:test-digest")); + assert!(debug.contains("additional-ca.crt")); + assert!(!debug.contains("not-secret-in-a-real-fixture")); + assert_eq!(bundle.pem(), pem); + assert_eq!(bundle.normalized_pem(), pem); + assert_eq!(bundle.certificate_count(), 1); + assert_eq!(bundle.digest(), "sha256:test-digest"); + assert_eq!( + bundle.artifact_path(), + Path::new("/var/lib/openshell/network/additional-ca.crt") + ); + assert!(!bundle.is_empty()); + } + + #[test] + fn cloned_bundles_share_immutable_material() { + let bundle = NetworkSupervisorTrustBundle::new( + b"certificate".to_vec(), + 1, + "sha256:test", + PathBuf::from("/tmp/additional-ca.crt"), + ); + assert_eq!(bundle.clone(), bundle); + } + + fn test_bundle(path: PathBuf, contents: &[u8]) -> NetworkSupervisorTrustBundle { + NetworkSupervisorTrustBundle::new(contents.to_vec(), 1, "sha256:test-generation", path) + } + + fn set_read_only(path: &Path) { + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o444)).unwrap(); + } + #[cfg(not(unix))] + let _ = path; + } + + #[cfg(unix)] + #[test] + fn artifact_verifier_rejects_writable_files() { + use std::os::unix::fs::PermissionsExt as _; + + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("additional-ca.crt"); + std::fs::write(&path, b"normalized test CA\n").unwrap(); + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o644)).unwrap(); + + let error = test_bundle(path, b"normalized test CA\n") + .verify_artifact() + .unwrap_err(); + assert!(matches!( + error, + NetworkSupervisorTrustArtifactError::InsecurePermissions { .. } + )); + assert!(!error.to_string().contains("normalized test CA")); + } + + #[test] + fn artifact_verifier_accepts_exact_file_contents() { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("additional-ca.crt"); + let contents = b"normalized test CA\n"; + std::fs::write(&path, contents).unwrap(); + set_read_only(&path); + + test_bundle(path, contents).verify_artifact().unwrap(); + } + + #[test] + fn artifact_verifier_rejects_replaced_contents_without_disclosing_them() { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("additional-ca.crt"); + let expected = b"expected normalized CA\n"; + let replacement = b"replacement private CA must not be disclosed\n"; + std::fs::write(&path, replacement).unwrap(); + set_read_only(&path); + + let error = test_bundle(path, expected).verify_artifact().unwrap_err(); + assert!(matches!( + error, + NetworkSupervisorTrustArtifactError::GenerationMismatch { .. } + )); + let diagnostic = error.to_string(); + assert!(diagnostic.contains("sha256:test-generation")); + assert!(!diagnostic.contains("expected normalized CA")); + assert!(!diagnostic.contains("replacement private CA")); + } + + #[test] + fn artifact_verifier_rejects_non_regular_files() { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("directory"); + std::fs::create_dir(&path).unwrap(); + + let error = test_bundle(path, b"contents") + .verify_artifact() + .unwrap_err(); + assert!(matches!( + error, + NetworkSupervisorTrustArtifactError::NotRegularFile { .. } + )); + } + + #[cfg(unix)] + #[test] + fn artifact_verifier_rejects_symlinks() { + use std::os::unix::fs::symlink; + + let temp = tempfile::tempdir().unwrap(); + let target = temp.path().join("target"); + let link = temp.path().join("additional-ca.crt"); + std::fs::write(&target, b"normalized test CA\n").unwrap(); + symlink(&target, &link).unwrap(); + + let error = test_bundle(link, b"normalized test CA\n") + .verify_artifact() + .unwrap_err(); + assert!(matches!( + error, + NetworkSupervisorTrustArtifactError::NotRegularFile { .. } + )); + } + + #[test] + fn artifact_verifier_enforces_shared_size_bound() { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("additional-ca.crt"); + let at_limit = vec![b'x'; MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES]; + std::fs::write(&path, &at_limit).unwrap(); + set_read_only(&path); + test_bundle(path.clone(), &at_limit) + .verify_artifact() + .unwrap(); + + let oversized = vec![b'x'; MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES + 1]; + std::fs::remove_file(&path).unwrap(); + std::fs::write(&path, &oversized).unwrap(); + set_read_only(&path); + let error = test_bundle(path, b"expected") + .verify_artifact() + .unwrap_err(); + assert!(matches!( + error, + NetworkSupervisorTrustArtifactError::TooLarge { limit, .. } + if limit == MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES + )); + assert!(!error.to_string().contains(&"x".repeat(64))); + } +} diff --git a/crates/openshell-crypto/README.md b/crates/openshell-crypto/README.md index 68efe366e..183b95b4b 100644 --- a/crates/openshell-crypto/README.md +++ b/crates/openshell-crypto/README.md @@ -140,6 +140,7 @@ not a whole-application crypto coverage claim: | Provider-refresh advisory-lock keys | Deterministic database lock selection for coordinating refreshes, not authentication or encryption. These remain outside the selected-backend contract. | | Endpoint IDs and pagination fingerprints | Deterministic identification/request matching, not authentication. These remain outside the selected-backend contract. | | Debug token fingerprints | Diagnostic display only; direct hashing remains outside the contract even though the input is a credential. | +| Destination-trust generation labels (`openshell-server` network trust, Podman operator staging, Kubernetes ConfigMap naming, VM staged-snapshot checks) | Deterministic identity derivation for naming and generation equality, not encryption or proof of authorization. Trust decisions use the gateway's normalization, immutable staging, exact-byte validation, and TLS verification rather than these digests. These retain `sha2` directly. | | Schema compatibility tests, content-cache identities, retry jitter, test fixtures | Non-authentication uses remain unchanged. | Cargo feature selection does not prevent a crate from adding another crypto diff --git a/crates/openshell-driver-docker/src/lib.rs b/crates/openshell-driver-docker/src/lib.rs index 3fc8c679b..970c8d4aa 100644 --- a/crates/openshell-driver-docker/src/lib.rs +++ b/crates/openshell-driver-docker/src/lib.rs @@ -58,7 +58,8 @@ use openshell_core::proto_struct::{ deserialize_optional_non_empty_string_list, struct_to_json_value, }; use openshell_core::{ - AppArmorProfile, Error, ImagePullPolicy, Result as CoreResult, UpstreamProxyConfig, + AppArmorProfile, Error, ImagePullPolicy, NetworkSupervisorTrustBundle, Result as CoreResult, + UpstreamProxyConfig, }; use openshell_isolation_interface::contract::ResolvedWorkloadIdentity; use openshell_sandbox_backend::boundary_protocol::{ @@ -138,6 +139,9 @@ const SUPERVISOR_AUTH_BUNDLE_FILE: &str = "supervisor-auth.json"; const START_GENERATION_FILE: &str = "start-generation"; const HOST_OPEN_SHELL_INTERNAL: &str = "host.openshell.internal"; const HOST_DOCKER_INTERNAL: &str = "host.docker.internal"; +/// Fixed in-container path for the gateway-owned additional destination trust. +/// It is mounted only into the separate trusted supervisor container. +const ADDITIONAL_CA_MOUNT_PATH: &str = openshell_core::driver_utils::ADDITIONAL_CA_MOUNT_PATH; fn provisioning_span( parent: &opentelemetry::Context, @@ -370,6 +374,9 @@ struct DockerDriverRuntimeConfig { proxy_ca_bundle: Option, provider_spiffe_workload_api_socket: Option, app_armor_profile: Option, + /// Immutable gateway-normalized additional destination trust. This is + /// deliberately separate from `DockerComputeConfig` and sandbox input. + network_trust: Option, } #[derive(Debug, Clone, Copy)] @@ -897,6 +904,7 @@ impl DockerComputeDriver { gateway_bind_address: SocketAddr, gateway_log_level: &str, docker_config: &DockerComputeConfig, + network_trust: Option, ) -> CoreResult { docker_config.validate_configuration(gateway_bind_address)?; let socket_path = docker_config @@ -1032,6 +1040,7 @@ impl DockerComputeDriver { .provider_spiffe_workload_api_socket .clone(), app_armor_profile: docker_config.app_armor_profile.clone(), + network_trust, }, events: broadcast::channel(WATCH_BUFFER).0, pending: Arc::new(Mutex::new(HashMap::new())), @@ -1555,6 +1564,7 @@ impl DockerComputeDriver { } async fn create_sandbox_inner(&self, sandbox: &DriverSandbox) -> Result<(), Status> { + self.verify_network_trust()?; let validated = Self::validated_sandbox(sandbox, &self.config)?; Self::validate_sandbox_auth(sandbox)?; self.validate_user_volume_mounts_available(&validated.driver_config, &sandbox.workspace) @@ -1633,6 +1643,7 @@ impl DockerComputeDriver { } } + /// Provision from gateway-owned immutable input. async fn provision_sandbox_inner( &self, sandbox: &DriverSandbox, @@ -1642,6 +1653,10 @@ impl DockerComputeDriver { DockerProvisioningFailure::new("ContainerCreateFailed", status.message()) })?; let template = validated.template; + let cleanup_runtime_volumes = || async { + let _ = + remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config).await; + }; let image = async { openshell_otel::record_error_result( self.ensure_image_available(&sandbox.id, &template.image) @@ -1668,7 +1683,7 @@ impl DockerComputeDriver { prepare_docker_boundary_state_dir(sandbox, &self.config).map_err(|status| { DockerProvisioningFailure::new("BoundaryStateCreateFailed", status.message()) })?; - create_docker_channel_volume(&self.docker, sandbox, &self.config) + create_docker_channel_volume(&self.docker, sandbox, &self.config, true) .await .map_err(|status| { cleanup_docker_boundary_state(sandbox, &self.config); @@ -1677,8 +1692,7 @@ impl DockerComputeDriver { let token_file_created = match write_sandbox_token_file(sandbox, &self.config).await { Ok(created) => created, Err(status) => { - let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config) - .await; + cleanup_runtime_volumes().await; cleanup_docker_boundary_state(sandbox, &self.config); return Err(DockerProvisioningFailure::new( "SandboxTokenWriteFailed", @@ -1687,8 +1701,7 @@ impl DockerComputeDriver { } }; if !token_file_created { - let _ = - remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config).await; + cleanup_runtime_volumes().await; cleanup_docker_boundary_state(sandbox, &self.config); return Err(DockerProvisioningFailure::new( "SandboxTokenWriteFailed", @@ -1707,8 +1720,7 @@ impl DockerComputeDriver { { Ok(devices) => devices, Err(status) => { - let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config) - .await; + cleanup_runtime_volumes().await; cleanup_docker_boundary_state(sandbox, &self.config); return Err(DockerProvisioningFailure::new( "ContainerCreateFailed", @@ -1726,8 +1738,7 @@ impl DockerComputeDriver { ) { Ok(body) => body, Err(status) => { - let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config) - .await; + cleanup_runtime_volumes().await; cleanup_docker_boundary_state(sandbox, &self.config); return Err(DockerProvisioningFailure::new( "ContainerCreateFailed", @@ -1781,8 +1792,7 @@ impl DockerComputeDriver { let created = match create_result { Ok(created) => created, Err(error) => { - let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config) - .await; + cleanup_runtime_volumes().await; cleanup_docker_boundary_state(sandbox, &self.config); return Err(DockerProvisioningFailure::from_status( "ContainerCreateFailed", @@ -1800,8 +1810,7 @@ impl DockerComputeDriver { Some(RemoveContainerOptionsBuilder::default().force(true).build()), ) .await; - let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config) - .await; + cleanup_runtime_volumes().await; cleanup_docker_boundary_state(sandbox, &self.config); return Err(DockerProvisioningFailure::from_status( "OuterFenceInspectFailed", @@ -1822,8 +1831,7 @@ impl DockerComputeDriver { Some(RemoveContainerOptionsBuilder::default().force(true).build()), ) .await; - let _ = - remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config).await; + cleanup_runtime_volumes().await; cleanup_docker_boundary_state(sandbox, &self.config); return Err(DockerProvisioningFailure::from_status( "OuterFenceRejected", @@ -1857,8 +1865,7 @@ impl DockerComputeDriver { Some(RemoveContainerOptionsBuilder::default().force(true).build()), ) .await; - let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config) - .await; + cleanup_runtime_volumes().await; cleanup_docker_boundary_state(sandbox, &self.config); return Err(DockerProvisioningFailure::new( "BoundaryConfigWriteFailed", @@ -1897,8 +1904,7 @@ impl DockerComputeDriver { ); } cleanup_docker_boundary_state(sandbox, &self.config); - let _ = - remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config).await; + cleanup_runtime_volumes().await; return Err(DockerProvisioningFailure::from_status( "ContainerStartFailed", create_status_from_docker_error("start docker sandbox container", err), @@ -1933,8 +1939,7 @@ impl DockerComputeDriver { Some(RemoveContainerOptionsBuilder::default().force(true).build()), ) .await; - let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config) - .await; + cleanup_runtime_volumes().await; cleanup_docker_boundary_state(sandbox, &self.config); return Err(DockerProvisioningFailure::new( supervisor_start_failure_reason(&status, "ControlSupervisorStartFailed"), @@ -1978,6 +1983,20 @@ impl DockerComputeDriver { self.runtime_failures.lock().await.remove(sandbox_id); } + /// Re-read the immutable artifact directly before any container launch or + /// stopped-pair reconciliation. The bundle never appears in a sandbox + /// request or workload environment, and errors redact its PEM contents. + fn verify_network_trust(&self) -> Result<(), Status> { + if let Some(bundle) = self.config.network_trust.as_ref() { + bundle.verify_artifact().map_err(|error| { + Status::failed_precondition(format!( + "verify gateway-owned network additional CA artifact: {error}" + )) + })?; + } + Ok(()) + } + fn control_failure_context( &self, sandbox: DriverSandbox, @@ -2076,6 +2095,9 @@ impl DockerComputeDriver { } async fn reconcile_runtime_resources_at_startup(&self) -> Result<(), Status> { + // Startup reconciliation must not inspect a changed artifact and then + // leave a trusted supervisor eligible for launch. + self.verify_network_trust()?; let sandboxes = self.list_managed_container_summaries().await?; let sandbox_ids = sandboxes .iter() @@ -2449,6 +2471,7 @@ impl DockerComputeDriver { sandbox_name: &str, generation_id: &str, launch_authentication: &[u8], + durable_sandbox: Option<&DriverSandbox>, ) -> Result { let span_status = openshell_otel::ErrorStatusGuard::current(); require_sandbox_identifier(sandbox_id, sandbox_name)?; @@ -2464,6 +2487,7 @@ impl DockerComputeDriver { sandbox_name, &generation, launch_authentication, + durable_sandbox, )) .await; self.lifecycle_event_fences.finish_start(sandbox_id); @@ -2476,7 +2500,9 @@ impl DockerComputeDriver { sandbox_name: &str, generation: &openshell_core::sandbox_generation::SandboxGenerationId, launch_authentication: &[u8], + _durable_sandbox: Option<&DriverSandbox>, ) -> Result { + self.verify_network_trust()?; let Some(container) = self .find_managed_container_summary(sandbox_id, sandbox_name) .await? @@ -2509,6 +2535,11 @@ impl DockerComputeDriver { .as_ref() .and_then(|labels| labels.get(LABEL_SANDBOX_ID)) .map_or(sandbox_id, String::as_str); + // No destination-trust gating here: the trusted supervisor container + // is re-created from the gateway's current configuration on every + // start (see `ensure_control_process_for_container`), so stop/start + // picks up the current additional CA bundle without ever touching + // the workload container or its data. if !container_state_needs_start(state) { adopt_or_verify_docker_start_generation(resolved_sandbox_id, &self.config, generation) .await?; @@ -3441,6 +3472,7 @@ impl ComputeDriver for DockerComputeDriver { &request.name, &request.generation_id, &request.launch_authentication, + request.sandbox.as_ref(), )) .await? { @@ -4165,6 +4197,7 @@ async fn create_docker_channel_volume( docker: &Docker, sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig, + create: bool, ) -> Result<(), Status> { let names = [ docker_channel_volume_name(sandbox, config), @@ -4173,7 +4206,7 @@ async fn create_docker_channel_volume( ]; for (index, name) in names.iter().enumerate() { if let Err(error) = - create_docker_runtime_volume(docker, sandbox, config, name.clone()).await + create_docker_runtime_volume(docker, sandbox, config, name.clone(), create).await { for created in &names[..index] { let _ = remove_docker_volume(docker, created).await; @@ -4189,6 +4222,7 @@ async fn create_docker_runtime_volume( sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig, name: String, + create: bool, ) -> Result<(), Status> { let expected_labels = HashMap::from([ ( @@ -4205,16 +4239,18 @@ async fn create_docker_runtime_volume( LABEL_ISOLATION_BACKEND_OPEN_SHELL.to_string(), ), ]); - docker - .create_volume(VolumeCreateRequest { - name: Some(name.clone()), - labels: Some(expected_labels.clone()), - ..Default::default() - }) - .await - .map_err(|error| { - Status::internal(format!("create Docker sandbox runtime volume: {error}")) - })?; + if create { + docker + .create_volume(VolumeCreateRequest { + name: Some(name.clone()), + labels: Some(expected_labels.clone()), + ..Default::default() + }) + .await + .map_err(|error| { + Status::internal(format!("create Docker sandbox runtime volume: {error}")) + })?; + } let volume = docker.inspect_volume(&name).await.map_err(|error| { Status::internal(format!("inspect Docker sandbox channel volume: {error}")) })?; @@ -4297,10 +4333,14 @@ async fn write_sandbox_token_file( let Some(spec) = sandbox.spec.as_ref() else { return Ok(false); }; - if spec.sandbox_token.is_empty() { - return Ok(false); - } let path = sandbox_token_host_path(sandbox, config)?; + if spec.sandbox_token.is_empty() { + // Restart reconciliation receives a durable public Sandbox, which + // intentionally excludes the bearer token. Its existing private token + // file remains the source of truth while the stopped workload is held + // as rollback state. + return Ok(path.is_file()); + } if let Some(parent) = path.parent() { openshell_core::paths::create_dir_restricted(parent).map_err(|err| { Status::internal(format!( @@ -4594,6 +4634,14 @@ async fn stage_docker_sandbox_bundle( Ok(()) } +/// Restore the old stopped workload's authenticated channel from the private +/// rollback state. This is used only after a replacement failure; its input is +/// driver-owned files saved before the mutation, never inspected container +/// configuration. +/// Stream a stopped workload's workspace through one bounded anonymous file. +/// Docker's archive endpoint returns an untrusted tar stream, so reject every +/// entry outside the original workspace and all non-file/non-directory entry +/// types before uploading the same filtered archive to the replacement. fn decode_docker_launch_authentication( encoded: &[u8], ) -> Result { @@ -5245,12 +5293,58 @@ fn docker_supervisor_host_address(grpc_endpoint: &str) -> Option { } } +fn docker_network_trust_supervisor_args( + bundle: Option<&NetworkSupervisorTrustBundle>, +) -> Vec { + bundle.map_or_else(Vec::new, |_bundle| { + vec![ + "--additional-ca-bundle".to_string(), + ADDITIONAL_CA_MOUNT_PATH.to_string(), + ] + }) +} + +fn docker_network_trust_supervisor_mount( + bundle: &NetworkSupervisorTrustBundle, +) -> Result { + bundle.verify_artifact().map_err(|error| { + Status::failed_precondition(format!( + "verify gateway-owned network additional CA artifact: {error}" + )) + })?; + let source = bundle.artifact_path().to_str().ok_or_else(|| { + Status::failed_precondition(format!( + "network additional CA artifact path is not valid UTF-8: {}", + bundle.artifact_path().display() + )) + })?; + driver_mounts::validate_absolute_mount_source(source, "network additional CA artifact") + .map_err(Status::failed_precondition)?; + Ok(Mount { + target: Some(ADDITIONAL_CA_MOUNT_PATH.to_string()), + source: Some(source.to_string()), + typ: Some(MountTypeEnum::BIND), + read_only: Some(true), + ..Default::default() + }) +} + async fn spawn_docker_control_process( docker: &Docker, sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig, failure_context: DockerRuntimeFailureContext, ) -> Result { + // This is the last gateway-side check before the trusted container gets a + // bind mount. It catches replacement/tamper after workload preparation + // without ever exposing the artifact to that workload. + if let Some(bundle) = config.network_trust.as_ref() { + bundle.verify_artifact().map_err(|error| { + Status::failed_precondition(format!( + "verify gateway-owned network additional CA artifact: {error}" + )) + })?; + } let (network_mode, supervisor_endpoint) = docker_supervisor_networking(config)?; info!(sandbox_id = %sandbox.id, ?network_mode, endpoint = %supervisor_endpoint, "Selected Docker supervisor networking"); @@ -5356,6 +5450,9 @@ async fn spawn_docker_control_process( )); let runtime_initiated = read_docker_runtime_descriptor(&sandbox.id, config).await? .is_some_and(|descriptor| matches!(descriptor.transport, openshell_sandbox_backend::boundary_protocol::SandboxTransport::RuntimeInitiatedUnix { .. })); + command.extend(docker_network_trust_supervisor_args( + config.network_trust.as_ref(), + )); let mut supervisor_mounts = vec![ Mount { target: Some(BOUNDARY_MOUNT_PATH.to_string()), @@ -5394,6 +5491,9 @@ async fn spawn_docker_control_process( }); } + if let Some(bundle) = config.network_trust.as_ref() { + supervisor_mounts.push(docker_network_trust_supervisor_mount(bundle)?); + } if let Some(socket) = config.provider_spiffe_workload_api_socket.as_ref() { let parent = socket.parent().ok_or_else(|| { Status::failed_precondition("provider SPIFFE socket has no parent directory") @@ -5795,6 +5895,9 @@ async fn stop_docker_control_process(mut process: DockerControlProcess) { let _ = process.task.await; } +/// Make a private, on-disk rollback copy of the driver state before a stopped +/// workload is replaced. Only regular files are copied: a compromised state +/// directory cannot smuggle symlinks into the gateway filesystem. fn cleanup_docker_boundary_state(sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig) { cleanup_docker_boundary_state_by_id(&sandbox.id, config); } diff --git a/crates/openshell-driver-docker/src/main.rs b/crates/openshell-driver-docker/src/main.rs index 153929471..d92004cdd 100644 --- a/crates/openshell-driver-docker/src/main.rs +++ b/crates/openshell-driver-docker/src/main.rs @@ -97,7 +97,7 @@ async fn main() -> Result<()> { schema_version: args.otlp_ocsf_schema_version.clone(), } }); - let driver = DockerComputeDriver::new(args.gateway_bind, &args.log_level, &docker_config) + let driver = DockerComputeDriver::new(args.gateway_bind, &args.log_level, &docker_config, None) .await .into_diagnostic()?; diff --git a/crates/openshell-driver-docker/src/tests.rs b/crates/openshell-driver-docker/src/tests.rs index 2fcc7b869..8c0bf9bc8 100644 --- a/crates/openshell-driver-docker/src/tests.rs +++ b/crates/openshell-driver-docker/src/tests.rs @@ -345,6 +345,7 @@ fn runtime_config() -> DockerDriverRuntimeConfig { proxy_ca_bundle: None, provider_spiffe_workload_api_socket: None, app_armor_profile: Some(AppArmorProfile::Unconfined), + network_trust: None, } } @@ -428,7 +429,7 @@ async fn docker_constructor_rejects_proxy_ca_bundle_without_proxy() { config.upstream_proxy.https_proxy = None; let Err(error) = - DockerComputeDriver::new("127.0.0.1:17670".parse().unwrap(), "info", &config).await + DockerComputeDriver::new("127.0.0.1:17670".parse().unwrap(), "info", &config, None).await else { panic!("constructor must reject incoherent proxy CA configuration before Docker I/O"); }; @@ -1037,6 +1038,7 @@ async fn start_sandbox_span_does_not_capture_launch_authentication() { "sandbox", "invalid-generation", b"secret-launch-authentication", + None, ) .with_subscriber(subscriber) .await @@ -1126,7 +1128,8 @@ async fn tracing_direct_start_exports_a_docker_start_span() { let driver = test_driver_with_config(runtime_config()); Box::pin( - DockerComputeDriver::start_sandbox(&driver, "", "", "", &[]).with_subscriber(subscriber), + DockerComputeDriver::start_sandbox(&driver, "", "", "", &[], None) + .with_subscriber(subscriber), ) .await .expect_err("missing identifier should fail"); @@ -2579,6 +2582,15 @@ fn build_container_create_body_replaces_inherited_cmd_with_sandbox_bootstrap() { BOUNDARY_CONFIG_MOUNT_PATH.to_string(), ]) ); + assert!( + create_body + .host_config + .as_ref() + .and_then(|config| config.binds.as_ref()) + .is_some_and(|binds| binds + .iter() + .all(|bind| !bind.contains(ADDITIONAL_CA_MOUNT_PATH))) + ); assert_eq!( create_body .labels @@ -2603,6 +2615,128 @@ fn build_container_create_body_replaces_inherited_cmd_with_sandbox_bootstrap() { assert!(create_body.networking_config.is_none()); } +#[test] +fn additional_destination_trust_is_supervisor_only_and_digest_paired() { + let dir = TempDir::new().unwrap(); + let artifact = dir.path().join("normalized-additional-ca.crt"); + let pem = b"normalized certificate fixture"; + fs::write(&artifact, pem).unwrap(); + #[cfg(unix)] + fs::set_permissions( + &artifact, + std::os::unix::fs::PermissionsExt::from_mode(0o444), + ) + .unwrap(); + let bundle = NetworkSupervisorTrustBundle::new( + pem.to_vec(), + 1, + "sha256:paired-digest", + artifact.clone(), + ); + let mut config = runtime_config(); + config.network_trust = Some(bundle.clone()); + + let workload = build_container_create_body(&test_sandbox(), &config).unwrap(); + assert!( + workload + .labels + .as_ref() + .and_then(|labels| labels.get(LABEL_SANDBOX_ID)) + .is_some() + ); + assert!( + workload + .cmd + .as_ref() + .unwrap() + .iter() + .all(|argument| argument != "--additional-ca-bundle") + ); + assert!( + workload + .host_config + .as_ref() + .and_then(|host| host.mounts.as_ref()) + .unwrap() + .iter() + .all(|mount| mount.target.as_deref() != Some(ADDITIONAL_CA_MOUNT_PATH)) + ); + + let args = docker_network_trust_supervisor_args(Some(&bundle)); + assert!( + args.windows(2) + .any(|arguments| { arguments == ["--additional-ca-bundle", ADDITIONAL_CA_MOUNT_PATH] }) + ); + let mount = docker_network_trust_supervisor_mount(&bundle).unwrap(); + assert_eq!(mount.target.as_deref(), Some(ADDITIONAL_CA_MOUNT_PATH)); + assert_eq!(mount.source.as_deref(), artifact.to_str()); + assert_eq!(mount.read_only, Some(true)); +} + +#[test] +fn tampered_additional_destination_trust_fails_closed_without_pem_contents() { + let dir = TempDir::new().unwrap(); + let artifact = dir.path().join("normalized-additional-ca.crt"); + fs::write(&artifact, b"expected").unwrap(); + #[cfg(unix)] + fs::set_permissions( + &artifact, + std::os::unix::fs::PermissionsExt::from_mode(0o444), + ) + .unwrap(); + let bundle = NetworkSupervisorTrustBundle::new( + b"expected".to_vec(), + 1, + "sha256:expected", + artifact.clone(), + ); + fs::remove_file(&artifact).unwrap(); + fs::write(&artifact, b"BEGIN CERTIFICATE modified secret data").unwrap(); + #[cfg(unix)] + fs::set_permissions( + &artifact, + std::os::unix::fs::PermissionsExt::from_mode(0o444), + ) + .unwrap(); + + let error = docker_network_trust_supervisor_mount(&bundle) + .expect_err("a changed gateway artifact must fail closed"); + assert!(error.message().contains("network additional CA artifact")); + assert!(!error.message().contains("modified secret data")); +} + +#[test] +fn user_mount_cannot_replace_network_trust_destination() { + let dir = TempDir::new().unwrap(); + let source = dir.path().join("user-ca.crt"); + fs::write(&source, b"user-controlled fixture").unwrap(); + let mut config = runtime_config(); + config.enable_bind_mounts = true; + let mut sandbox = test_sandbox(); + sandbox + .spec + .as_mut() + .unwrap() + .template + .as_mut() + .unwrap() + .driver_config = Some(json_struct(serde_json::json!({ + "mounts": [{ + "type": "bind", + "source": source, + "target": ADDITIONAL_CA_MOUNT_PATH + }] + }))); + + let error = build_container_create_body(&sandbox, &config) + .expect_err("user mount must not mask network trust material"); + assert_eq!(error.code(), tonic::Code::FailedPrecondition); + assert!( + error.message().contains("reserved OpenShell path"), + "unexpected error: {error}" + ); +} + #[test] fn validate_sandbox_rejects_gpu_when_cdi_unavailable() { let config = runtime_config(); diff --git a/crates/openshell-driver-kubernetes/Cargo.toml b/crates/openshell-driver-kubernetes/Cargo.toml index f1d55cc0f..1fc7c7d00 100644 --- a/crates/openshell-driver-kubernetes/Cargo.toml +++ b/crates/openshell-driver-kubernetes/Cargo.toml @@ -36,6 +36,7 @@ k8s-openapi = { workspace = true } tower = { workspace = true, features = ["timeout"] } serde = { workspace = true } serde_json = { workspace = true } +sha2 = { workspace = true } clap = { workspace = true } tracing = { workspace = true } tracing-subscriber = { workspace = true } diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index 91a1ff416..cd7532835 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -31,8 +31,8 @@ use k8s_openapi::api::authentication::v1::{ TokenReview, TokenReviewSpec, TokenReviewStatus, UserInfo, }; use k8s_openapi::api::core::v1::{ - Event as KubeEventObj, Namespace, Node, PersistentVolumeClaimVolumeSource, Pod, Secret, - Service, ServiceAccount, Volume, VolumeMount, + ConfigMap, Event as KubeEventObj, Namespace, Node, PersistentVolumeClaimVolumeSource, Pod, + Secret, Service, ServiceAccount, Volume, VolumeMount, }; use k8s_openapi::api::networking::v1::{ NetworkPolicy, NetworkPolicyIngressRule, NetworkPolicyPeer, NetworkPolicyPort, @@ -49,6 +49,7 @@ use kube::runtime::WatchStreamExt; use kube::runtime::wait::await_condition; use kube::runtime::watcher::{self, Event}; use kube::{Client, Error as KubeError}; +use openshell_core::NetworkSupervisorTrustBundle; use openshell_core::driver_mounts; use openshell_core::driver_utils::{ LABEL_GATEWAY_ID, LABEL_MANAGED_BY, LABEL_MANAGED_BY_VALUE, LABEL_SANDBOX_ID, @@ -79,6 +80,7 @@ use openshell_supervisor_control::{ RegistrationTraceContext, SandboxInstanceRegistration, SharedSupervision, }; use serde::Deserialize; +use sha2::{Digest as _, Sha256}; use std::collections::{BTreeMap, HashSet}; use std::path::{Path, PathBuf}; use std::pin::Pin; @@ -125,6 +127,194 @@ const ANNOTATION_SANDBOX_RUNTIME_CONTROL_POLICY_UID: &str = const ANNOTATION_SANDBOX_RUNTIME_CONTROL_POLICY_GENERATION: &str = "openshell.ai/sandbox-runtime-control-policy-generation"; +/// Full gateway-normalized destination-trust digest recorded on each immutable +/// `ConfigMap` generation. The digest is intentionally not truncated here even +/// though the object name uses stable, DNS-safe hash prefixes. +const ANNOTATION_ADDITIONAL_CA_GENERATION: &str = "openshell.ai/additional-ca-generation"; +const ADDITIONAL_CA_CONFIG_MAP_PREFIX: &str = "openshell-additional-ca-"; +const ADDITIONAL_CA_DATA_KEY: &str = "ca.crt"; + +/// A validated immutable `ConfigMap` generation selected for a supervisor Pod. +/// This contains only an object name and non-secret digest; PEM bytes remain in +/// the API object and are never copied into a workload template or Secret. +#[derive(Clone, Debug, PartialEq, Eq)] +struct NetworkAdditionalCaConfigMap { + name: String, + digest: String, +} + +fn network_additional_ca_config_map_name(gateway_id: &str, digest: &str) -> String { + // Kubernetes labels have a 63-byte limit. Keep recognizable stable + // prefixes while hashing both variable inputs. The complete digest is + // retained in an annotation and checked before every supervisor creation. + // The hashes derive deterministic object names, not authenticity proofs; + // trust comes from the immutable flag, the digest annotation, and the + // exact-byte validation. Classified as a non-security identity hash in + // openshell-crypto/README.md. + let gateway_hash = format!("{:x}", Sha256::digest(gateway_id.as_bytes())); + let digest_hash = format!("{:x}", Sha256::digest(digest.as_bytes())); + format!( + "{ADDITIONAL_CA_CONFIG_MAP_PREFIX}{}-{}", + &gateway_hash[..12], + &digest_hash[..26], + ) +} + +fn network_additional_ca_config_map( + namespace: &str, + gateway_id: &str, + bundle: &NetworkSupervisorTrustBundle, +) -> Result { + let pem = std::str::from_utf8(bundle.normalized_pem()).map_err(|_| { + KubernetesDriverError::Precondition( + "gateway-normalized network additional CA bundle is not UTF-8".to_string(), + ) + })?; + let name = network_additional_ca_config_map_name(gateway_id, bundle.digest()); + Ok(ConfigMap { + metadata: ObjectMeta { + name: Some(name), + namespace: Some(namespace.to_string()), + labels: Some(BTreeMap::from([ + ( + LABEL_MANAGED_BY.to_string(), + LABEL_MANAGED_BY_VALUE.to_string(), + ), + (LABEL_GATEWAY_ID.to_string(), gateway_id.to_string()), + ])), + annotations: Some(BTreeMap::from([( + ANNOTATION_ADDITIONAL_CA_GENERATION.to_string(), + bundle.digest().to_string(), + )])), + // Generations are intentionally retained across Pod and Sandbox + // replacement. A sandbox owner reference would allow garbage + // collection to remove material still needed by a stopped PVC. + owner_references: None, + ..Default::default() + }, + immutable: Some(true), + data: Some(BTreeMap::from([( + ADDITIONAL_CA_DATA_KEY.to_string(), + pem.to_string(), + )])), + binary_data: None, + }) +} + +fn validate_network_additional_ca_config_map( + existing: &ConfigMap, + expected: &ConfigMap, +) -> Result<(), KubernetesDriverError> { + let name = expected.metadata.name.as_deref().unwrap_or_default(); + // Validate the gateway-owned label pairs individually. A mutating + // admission webhook may attach organizational labels to the created + // ConfigMap; an exact-map comparison would then reject the correct map on + // the create response and on every later create/start through this + // existing-object path, blocking sandbox launches despite the required + // ownership labels, immutable flag, digest, and certificate bytes + // remaining correct. Unrelated labels are therefore allowed; each + // gateway-owned pair must match exactly. + if let Some(expected_labels) = expected.metadata.labels.as_ref() { + let existing_labels = existing.metadata.labels.as_ref().ok_or_else(|| { + KubernetesDriverError::Precondition(format!( + "network additional CA ConfigMap {name} is not owned by this gateway" + )) + })?; + for (key, value) in expected_labels { + if existing_labels.get(key) != Some(value) { + return Err(KubernetesDriverError::Precondition(format!( + "network additional CA ConfigMap {name} is not owned by this gateway" + ))); + } + } + } + // The immutable generation must outlive individual sandbox objects. + // Reject even an explicitly empty ownerReferences field so only the + // API representation produced by our ownerless template is accepted. + if existing.metadata.owner_references.is_some() { + return Err(KubernetesDriverError::Precondition(format!( + "network additional CA ConfigMap {name} is not owned by this gateway" + ))); + } + let expected_digest = expected + .metadata + .annotations + .as_ref() + .and_then(|annotations| annotations.get(ANNOTATION_ADDITIONAL_CA_GENERATION)); + let actual_digest = existing + .metadata + .annotations + .as_ref() + .and_then(|annotations| annotations.get(ANNOTATION_ADDITIONAL_CA_GENERATION)); + if actual_digest != expected_digest { + return Err(KubernetesDriverError::Precondition(format!( + "network additional CA ConfigMap {name} has an unexpected trust generation" + ))); + } + if existing.immutable != Some(true) { + return Err(KubernetesDriverError::Precondition(format!( + "network additional CA ConfigMap {name} is not immutable" + ))); + } + if existing.data != expected.data || existing.binary_data.is_some() { + return Err(KubernetesDriverError::Precondition(format!( + "network additional CA ConfigMap {name} did not retain the exact gateway-normalized trust bundle" + ))); + } + Ok(()) +} + +async fn create_or_validate_network_additional_ca_config_map( + config_maps: &Api, + expected: &ConfigMap, +) -> Result<(), KubernetesDriverError> { + let name = expected.metadata.name.as_deref().ok_or_else(|| { + KubernetesDriverError::Message("network additional CA ConfigMap has no name".to_string()) + })?; + match tokio::time::timeout(KUBE_API_TIMEOUT, config_maps.get_opt(name)).await { + Ok(Ok(Some(existing))) => { + return validate_network_additional_ca_config_map(&existing, expected); + } + Ok(Ok(None)) => {} + Ok(Err(error)) => return Err(KubernetesDriverError::from_kube(error)), + Err(_) => { + return Err(KubernetesDriverError::Message( + "timed out reading network additional CA ConfigMap".to_string(), + )); + } + } + + match tokio::time::timeout( + KUBE_API_TIMEOUT, + config_maps.create(&PostParams::default(), expected), + ) + .await + { + // Validate the server response as well: an admission controller must + // not be able to silently alter the immutable material we mount. + Ok(Ok(created)) => validate_network_additional_ca_config_map(&created, expected), + // A concurrent gateway instance may win creation. Re-read exactly the + // named object and subject the winner to the same byte-for-byte checks; + // no LIST, WATCH, PATCH, UPDATE, or DELETE is ever used for this API. + Ok(Err(KubeError::Api(error))) if error.code == 409 => { + let existing = tokio::time::timeout(KUBE_API_TIMEOUT, config_maps.get(name)) + .await + .map_err(|_| { + KubernetesDriverError::Message( + "timed out validating concurrent network additional CA ConfigMap creation" + .to_string(), + ) + })? + .map_err(KubernetesDriverError::from_kube)?; + validate_network_additional_ca_config_map(&existing, expected) + } + Ok(Err(error)) => Err(KubernetesDriverError::from_kube(error)), + Err(_) => Err(KubernetesDriverError::Message( + "timed out creating network additional CA ConfigMap".to_string(), + )), + } +} + #[derive(Clone, Copy, Debug, PartialEq, Eq)] enum SandboxRuntimeBootstrapPhase { Preparing, @@ -894,6 +1084,10 @@ pub struct KubernetesComputeDriver { sandbox_api_version: Arc>, lifecycle_gates: Arc, config: KubernetesComputeConfig, + /// Gateway-normalized destination trust available only to this in-process + /// driver. The external driver protocol deliberately has no transport for + /// the PEM bytes or a gateway-local artifact path. + network_trust_bundle: Option, operator_allowlist: Option, /// Desired shared sandbox instances while new generations use the gateway /// topology. @@ -945,6 +1139,7 @@ impl KubernetesComputeDriver { sandbox_api_version: Arc::new(OnceCell::new()), lifecycle_gates: Arc::default(), config, + network_trust_bundle: None, operator_allowlist: None, shared_supervisor: None, } @@ -957,6 +1152,7 @@ impl KubernetesComputeDriver { pub async fn new( config: KubernetesComputeConfig, shutdown_rx: tokio::sync::watch::Receiver, + network_trust_bundle: Option, ) -> Result { if config.shared_supervisor.is_some() { return Err(KubernetesDriverError::Precondition( @@ -964,7 +1160,7 @@ impl KubernetesComputeDriver { .to_string(), )); } - Self::build(config, shutdown_rx, None).await + Self::build(config, shutdown_rx, None, network_trust_bundle).await } /// A driver whose new generations become sandbox instances on the shared @@ -984,13 +1180,17 @@ impl KubernetesComputeDriver { .to_string(), )); } - Self::build(config, shutdown_rx, Some(supervision)).await + // The gateway refuses the additional destination CA under the shared + // supervisor topology fail-closed, so no trust bundle reaches this + // constructor. + Self::build(config, shutdown_rx, Some(supervision), None).await } async fn build( config: KubernetesComputeConfig, shutdown_rx: tokio::sync::watch::Receiver, supervision: Option, + network_trust_bundle: Option, ) -> Result { config .resource_admission @@ -1090,6 +1290,7 @@ impl KubernetesComputeDriver { sandbox_api_version: Arc::new(OnceCell::new()), lifecycle_gates: Arc::default(), config, + network_trust_bundle, operator_allowlist, shared_supervisor, }; @@ -2652,6 +2853,29 @@ impl KubernetesComputeDriver { Ok(()) } + /// Create or validate the gateway-owned destination-trust generation for + /// one target namespace. The source PEM never crosses the external driver + /// boundary: only this in-process driver receives the normalized bundle. + async fn ensure_network_additional_ca_config_map( + &self, + namespace: &str, + ) -> Result, KubernetesDriverError> { + let Some(bundle) = self.network_trust_bundle.as_ref() else { + return Ok(None); + }; + let expected = + network_additional_ca_config_map(namespace, &self.config.gateway_id, bundle)?; + let name = expected.metadata.name.clone().ok_or_else(|| { + KubernetesDriverError::Message( + "network additional CA ConfigMap has no name".to_string(), + ) + })?; + let digest = bundle.digest().to_string(); + let config_maps = Api::::namespaced(self.client.clone(), namespace); + create_or_validate_network_additional_ca_config_map(&config_maps, &expected).await?; + Ok(Some(NetworkAdditionalCaConfigMap { name, digest })) + } + async fn wait_for_bootstrap_workload_pod( &self, pods: &Api, @@ -3121,6 +3345,12 @@ impl KubernetesComputeDriver { log_level: &str, upstream_proxy_ca_bundle_staged: bool, ) -> Result { + // Validate or create the immutable trust generation before creating + // the dedicated supervisor Pod. A same-name delete/recreate with + // altered bytes therefore fails closed while no control process runs. + let network_additional_ca = self + .ensure_network_additional_ca_config_map(namespace) + .await?; let pods: Api = Api::namespaced(self.client.clone(), namespace); pods.create( &PostParams::default(), @@ -3155,6 +3385,9 @@ impl KubernetesComputeDriver { .provider_spiffe_workload_api_socket_path .as_str(), ), + network_additional_ca + .as_ref() + .map(|generation| (generation.name.as_str(), generation.digest.as_str())), owner, &self.supervisor_otlp_environment(), ) @@ -3574,6 +3807,12 @@ impl KubernetesComputeDriver { vec![workload_owner, supervisor_owner], ) .await?; + // A restarted generation must revalidate immediately before + // resuming the gated supervisor as well as before creating + // it. The shared topology never carries an additional CA: + // the gateway refuses that combination fail-closed. + self.ensure_network_additional_ca_config_map(namespace) + .await?; pods.patch( &names.supervisor_pod, &PatchParams::default(), @@ -3930,6 +4169,12 @@ impl KubernetesComputeDriver { let names = SandboxRuntimeNames::for_generation(sandbox_id, generation.as_str()); self.create_sandbox_runtime_fence(&namespace, &names) .await?; + // Validate the current gateway generation before touching a stopped + // sandbox's stale companion. This detects equal-generation + // delete/recreate tampering before a supervisor can be created or + // released, while running supervisors remain untouched. + self.ensure_network_additional_ca_config_map(&namespace) + .await?; let mut stale_generations = sandbox_runtime_generation(&object).as_slice().to_vec(); stale_generations.push(generation.as_str()); self.delete_sandbox_runtime_supervisor(sandbox_id, &namespace, &stale_generations) @@ -8964,6 +9209,7 @@ mod tests { sandbox_api_version: Arc::new(OnceCell::new()), lifecycle_gates: Arc::default(), config: KubernetesComputeConfig::default(), + network_trust_bundle: None, operator_allowlist: None, shared_supervisor: None, }; @@ -8977,6 +9223,7 @@ mod tests { assert!(steps.lock().unwrap().is_empty()); } + use crate::sandbox_runtime::{ADDITIONAL_CA_MOUNT_PATH, ADDITIONAL_CA_VOLUME_NAME}; use openshell_core::progress::{ PROGRESS_ACTIVE_DETAIL_KEY, PROGRESS_ACTIVE_STEP_KEY, PROGRESS_COMPLETE_LABEL_KEY, PROGRESS_COMPLETE_STEP_KEY, @@ -9092,6 +9339,179 @@ mod tests { ) } + fn network_trust_bundle(gateway_id: &str, digest: &str) -> NetworkSupervisorTrustBundle { + NetworkSupervisorTrustBundle::new( + b"-----BEGIN CERTIFICATE-----\nfixture\n-----END CERTIFICATE-----\n".to_vec(), + 1, + digest, + PathBuf::from(format!("/gateway-state/{gateway_id}.crt")), + ) + } + + #[test] + fn destination_trust_config_map_is_immutable_content_addressed_and_retained() { + let digest = "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"; + let bundle = network_trust_bundle("gateway-a", digest); + let expected = network_additional_ca_config_map("workspace-a", "gateway-a", &bundle) + .expect("normalized PEM must be valid UTF-8"); + let name = expected.metadata.name.as_deref().expect("ConfigMap name"); + assert_eq!(name.len(), 63); + assert!(name.starts_with(ADDITIONAL_CA_CONFIG_MAP_PREFIX)); + assert!( + name.bytes() + .all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'-') + ); + assert_eq!(expected.immutable, Some(true)); + assert_eq!(expected.metadata.owner_references, None); + assert_eq!( + expected.metadata.labels.as_ref(), + Some(&BTreeMap::from([ + ( + LABEL_MANAGED_BY.to_string(), + LABEL_MANAGED_BY_VALUE.to_string() + ), + (LABEL_GATEWAY_ID.to_string(), "gateway-a".to_string()), + ])) + ); + assert_eq!( + expected + .metadata + .annotations + .as_ref() + .and_then(|annotations| annotations.get(ANNOTATION_ADDITIONAL_CA_GENERATION)), + Some(&digest.to_string()) + ); + assert_eq!( + expected.data.as_ref(), + Some(&BTreeMap::from([( + ADDITIONAL_CA_DATA_KEY.to_string(), + String::from_utf8(bundle.normalized_pem().to_vec()).unwrap(), + )])) + ); + assert_eq!(expected.binary_data, None); + assert_ne!( + name, + network_additional_ca_config_map_name("gateway-b", digest), + "gateway identity participates in content addressing" + ); + assert_ne!( + name, + network_additional_ca_config_map_name("gateway-a", "sha256:other-generation"), + "trust generation participates in content addressing" + ); + assert!(validate_network_additional_ca_config_map(&expected, &expected).is_ok()); + } + + #[test] + fn destination_trust_config_map_rejects_same_name_recreation_tampering() { + let bundle = network_trust_bundle( + "gateway-a", + "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + ); + let expected = network_additional_ca_config_map("workspace-a", "gateway-a", &bundle) + .expect("normalized PEM must be valid UTF-8"); + + let mut replaced = expected.clone(); + replaced.data = Some(BTreeMap::from([( + ADDITIONAL_CA_DATA_KEY.to_string(), + "not-a-certificate\n".to_string(), + )])); + assert!(matches!( + validate_network_additional_ca_config_map(&replaced, &expected), + Err(KubernetesDriverError::Precondition(message)) + if message.contains("did not retain the exact gateway-normalized trust bundle") + )); + + let mut binary_data = expected.clone(); + binary_data.binary_data = Some(BTreeMap::new()); + assert!(matches!( + validate_network_additional_ca_config_map(&binary_data, &expected), + Err(KubernetesDriverError::Precondition(message)) + if message.contains("did not retain the exact gateway-normalized trust bundle") + )); + + let mut wrong_gateway = expected.clone(); + wrong_gateway + .metadata + .labels + .as_mut() + .unwrap() + .insert(LABEL_GATEWAY_ID.to_string(), "other-gateway".to_string()); + assert!(matches!( + validate_network_additional_ca_config_map(&wrong_gateway, &expected), + Err(KubernetesDriverError::Precondition(message)) + if message.contains("is not owned by this gateway") + )); + + let mut sandbox_owned = expected.clone(); + sandbox_owned.metadata.owner_references = + Some(vec![sandbox_runtime_sandbox_owner_reference( + "sandbox", + "sandbox-uid", + "agents.x-k8s.io/v1beta1", + false, + )]); + assert!(matches!( + validate_network_additional_ca_config_map(&sandbox_owned, &expected), + Err(KubernetesDriverError::Precondition(message)) + if message.contains("is not owned by this gateway") + )); + + let mut stale_digest = expected.clone(); + stale_digest.metadata.annotations.as_mut().unwrap().insert( + ANNOTATION_ADDITIONAL_CA_GENERATION.to_string(), + "sha256:stale".to_string(), + ); + assert!(matches!( + validate_network_additional_ca_config_map(&stale_digest, &expected), + Err(KubernetesDriverError::Precondition(message)) + if message.contains("unexpected trust generation") + )); + } + + #[test] + fn destination_trust_config_map_allows_admission_added_labels() { + // A mutating admission webhook may attach organizational labels to + // the created ConfigMap. The gateway-owned pairs must still match + // exactly, but unrelated labels must not block every later + // create/start through the existing-object validation path. + let bundle = network_trust_bundle( + "gateway-a", + "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + ); + let expected = network_additional_ca_config_map("workspace-a", "gateway-a", &bundle) + .expect("normalized PEM must be valid UTF-8"); + + let mut admitted = expected.clone(); + admitted + .metadata + .labels + .as_mut() + .expect("expected ConfigMap has gateway-owned labels") + .insert( + "org.example/compliance-scan".to_string(), + "managed".to_string(), + ); + assert!( + validate_network_additional_ca_config_map(&admitted, &expected).is_ok(), + "an admission-added organizational label must not reject a correct \ + ConfigMap" + ); + + let mut missing_gateway_label = expected.clone(); + missing_gateway_label + .metadata + .labels + .as_mut() + .unwrap() + .remove(&LABEL_GATEWAY_ID.to_string()); + assert!(matches!( + validate_network_additional_ca_config_map(&missing_gateway_label, &expected), + Err(KubernetesDriverError::Precondition(message)) + if message.contains("is not owned by this gateway") + )); + } + #[test] fn boundary_authority_uses_stable_service_dns_name() { let names = SandboxRuntimeNames::new("sandbox-1"); @@ -10048,6 +10468,7 @@ mod tests { config: KubernetesComputeConfig::default(), operator_allowlist: None, shared_supervisor: None, + network_trust_bundle: None, }; driver .sandbox_api_version @@ -10140,6 +10561,7 @@ mod tests { config: KubernetesComputeConfig::default(), operator_allowlist: None, shared_supervisor: None, + network_trust_bundle: None, }; driver @@ -11421,6 +11843,33 @@ mod tests { ); let volumes = pod_template["spec"]["volumes"].as_array().unwrap(); + assert!( + !volumes + .iter() + .any(|volume| volume["name"] == ADDITIONAL_CA_VOLUME_NAME), + "destination trust is exclusive to the separate supervisor Pod" + ); + for container in pod_template["spec"]["containers"] + .as_array() + .into_iter() + .flatten() + .chain( + pod_template["spec"]["initContainers"] + .as_array() + .into_iter() + .flatten(), + ) + { + assert!( + !container["volumeMounts"].as_array().is_some_and(|mounts| { + mounts.iter().any(|mount| { + mount["name"] == ADDITIONAL_CA_VOLUME_NAME + || mount["mountPath"] == ADDITIONAL_CA_MOUNT_PATH + }) + }), + "destination trust leaked into a workload or init container: {container:#?}" + ); + } let pod_identity = volumes .iter() .find(|volume| volume["name"] == SANDBOX_POD_IDENTITY_VOLUME_NAME) @@ -12878,6 +13327,7 @@ mod tests { sandbox_api_version: Arc::new(OnceCell::new()), lifecycle_gates: Arc::default(), config, + network_trust_bundle: None, operator_allowlist: None, shared_supervisor: None, }; @@ -13183,6 +13633,7 @@ mod tests { sandbox_api_version: Arc::new(OnceCell::new()), lifecycle_gates: Arc::default(), config: managed_config(&[]), + network_trust_bundle: None, operator_allowlist: None, shared_supervisor: None, }; diff --git a/crates/openshell-driver-kubernetes/src/lifecycle_tests.rs b/crates/openshell-driver-kubernetes/src/lifecycle_tests.rs index d8ec7791c..3b9eea6dd 100644 --- a/crates/openshell-driver-kubernetes/src/lifecycle_tests.rs +++ b/crates/openshell-driver-kubernetes/src/lifecycle_tests.rs @@ -35,6 +35,7 @@ mod lifecycle_reconciliation { sandbox_api_version: Arc::new(OnceCell::new()), lifecycle_gates: Arc::default(), config: KubernetesComputeConfig::default(), + network_trust_bundle: None, operator_allowlist: None, shared_supervisor: None, }; diff --git a/crates/openshell-driver-kubernetes/src/main.rs b/crates/openshell-driver-kubernetes/src/main.rs index 209d28637..ec768b9ed 100644 --- a/crates/openshell-driver-kubernetes/src/main.rs +++ b/crates/openshell-driver-kubernetes/src/main.rs @@ -329,6 +329,7 @@ async fn main() -> Result<()> { sandbox_gid: args.sandbox_gid, }, shutdown_rx, + None, ) .await .into_diagnostic()?; diff --git a/crates/openshell-driver-kubernetes/src/sandbox_runtime.rs b/crates/openshell-driver-kubernetes/src/sandbox_runtime.rs index 004aacfb1..111b6ba7e 100644 --- a/crates/openshell-driver-kubernetes/src/sandbox_runtime.rs +++ b/crates/openshell-driver-kubernetes/src/sandbox_runtime.rs @@ -8,8 +8,8 @@ use std::path::Path; use k8s_openapi::ByteString; use k8s_openapi::api::core::v1::{ - CSIVolumeSource, Capabilities, Container, EmptyDirVolumeSource, EnvVar, KeyToPath, - LocalObjectReference, Pod, PodSchedulingGate, PodSecurityContext, PodSpec, Probe, + CSIVolumeSource, Capabilities, ConfigMapVolumeSource, Container, EmptyDirVolumeSource, EnvVar, + KeyToPath, LocalObjectReference, Pod, PodSchedulingGate, PodSecurityContext, PodSpec, Probe, ProjectedVolumeSource, Secret, SecretVolumeSource, SecurityContext, Service, ServiceAccountTokenProjection, ServicePort, ServiceSpec, TCPSocketAction, Volume, VolumeMount, VolumeProjection, @@ -67,6 +67,12 @@ pub const CONTROL_HEALTH_SOCKET_PATH: &str = "/run/openshell/health.sock"; /// Kubelet `tcpSocket` readiness port. An exec probe would start a supervisor /// process in every sandbox on every period. pub const CONTROL_HEALTH_PORT: u16 = 5501; + +/// `ConfigMap` volume name reserved for gateway-normalized destination trust. +pub const ADDITIONAL_CA_VOLUME_NAME: &str = "openshell-additional-ca"; +/// The only destination-trust path exposed to the dedicated supervisor Pod. +pub const ADDITIONAL_CA_MOUNT_PATH: &str = openshell_core::driver_utils::ADDITIONAL_CA_MOUNT_PATH; +const ADDITIONAL_CA_KEY: &str = "ca.crt"; pub const NAMESPACE_WORKLOAD_POLICY_NAME: &str = "openshell-sandbox-workloads"; pub const NAMESPACE_SUPERVISOR_EGRESS_POLICY_NAME: &str = "openshell-sandbox-supervisors"; pub const SUPERVISOR_TERMINATION_GRACE_PERIOD_SECONDS: i64 = 30; @@ -341,6 +347,10 @@ pub fn supervisor_pod( proxy_connect_by_hostname: bool, upstream_proxy_ca_bundle_staged: bool, provider_spiffe_socket_path: Option<&str>, + // Name and expected full digest of a gateway-normalized destination trust + // generation. The ConfigMap is mounted only in this dedicated control Pod; + // it is deliberately absent from the workload Pod and its bootstrap Secret. + network_additional_ca: Option<(&str, &str)>, owner: OwnerReference, tracing_environment: &[(&str, String)], ) -> Result { @@ -481,6 +491,22 @@ pub fn supervisor_pod( UPSTREAM_PROXY_CA_BUNDLE_PATH.to_string(), ]); } + if let Some((config_map_name, _digest)) = network_additional_ca { + command.extend([ + "--additional-ca-bundle".to_string(), + ADDITIONAL_CA_MOUNT_PATH.to_string(), + ]); + let mut mount = volume_mount(ADDITIONAL_CA_VOLUME_NAME, ADDITIONAL_CA_MOUNT_PATH, true); + // A subPath exposes `ca.crt` at the exact protected file path rather + // than mounting a ConfigMap directory over the supervisor's filesystem. + mount.sub_path = Some(ADDITIONAL_CA_KEY.to_string()); + volume_mounts.push(mount); + volumes.push(config_map_volume( + ADDITIONAL_CA_VOLUME_NAME, + config_map_name, + ADDITIONAL_CA_KEY, + )); + } if let Some((secret_name, secret_key)) = proxy_auth_secret { let auth_path = Path::new(openshell_core::container_paths::UPSTREAM_PROXY_AUTH_MOUNT_PATH); let mount_path = auth_path @@ -828,6 +854,23 @@ fn volume_mount(name: &str, mount_path: &str, read_only: bool) -> VolumeMount { } } +fn config_map_volume(name: &str, config_map_name: &str, key: &str) -> Volume { + Volume { + name: name.to_string(), + config_map: Some(ConfigMapVolumeSource { + name: config_map_name.to_string(), + default_mode: Some(0o444), + items: Some(vec![KeyToPath { + key: key.to_string(), + path: key.to_string(), + ..Default::default() + }]), + ..Default::default() + }), + ..Default::default() + } +} + fn secret_volume(name: &str, secret_name: &str, item: Option) -> Volume { Volume { name: name.to_string(), @@ -999,6 +1042,7 @@ mod tests { false, false, None, + None, owner(), &[], ) @@ -1058,6 +1102,7 @@ mod tests { false, false, None, + None, owner(), &[ ( @@ -1184,6 +1229,7 @@ mod tests { false, false, None, + None, owner(), &[], ) @@ -1323,6 +1369,88 @@ mod tests { assert_eq!(mount.read_only, Some(true)); } + #[test] + fn destination_trust_is_mounted_only_into_the_dedicated_supervisor_pod() { + let names = SandboxRuntimeNames::new("pair"); + let digest = "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"; + let pod = supervisor_pod( + "sandbox", + &names, + "pair", + "demo", + "gateway", + "supervisor:latest", + None, + "sandbox-sa", + 1000, + 1000, + &[], + "https://gateway:8080", + SupervisorClientTls::Disabled, + "{}", + "info", + 600, + None, + None, + None, + false, + false, + false, + None, + Some(("openshell-additional-ca-gateway-generation", digest)), + owner(), + &[], + ) + .expect("render supervisor Pod with destination trust"); + let spec = pod.spec.expect("Pod spec"); + let container = &spec.containers[0]; + assert!(container.command.as_deref().is_some_and(|command| { + command + .windows(2) + .any(|args| args == ["--additional-ca-bundle", ADDITIONAL_CA_MOUNT_PATH]) + })); + assert_eq!( + container + .volume_mounts + .as_deref() + .unwrap_or_default() + .iter() + .filter(|mount| mount.name == ADDITIONAL_CA_VOLUME_NAME) + .collect::>() + .as_slice(), + &[&VolumeMount { + name: ADDITIONAL_CA_VOLUME_NAME.to_string(), + mount_path: ADDITIONAL_CA_MOUNT_PATH.to_string(), + read_only: Some(true), + sub_path: Some(ADDITIONAL_CA_KEY.to_string()), + ..Default::default() + }] + ); + let volume = spec + .volumes + .as_deref() + .unwrap_or_default() + .iter() + .find(|volume| volume.name == ADDITIONAL_CA_VOLUME_NAME) + .expect("destination trust ConfigMap volume"); + assert_eq!( + volume + .config_map + .as_ref() + .map(|source| source.name.as_str()), + Some("openshell-additional-ca-gateway-generation") + ); + assert_eq!( + volume + .config_map + .as_ref() + .and_then(|source| source.items.as_deref()) + .and_then(|items| items.first()) + .map(|item| (item.key.as_str(), item.path.as_str())), + Some((ADDITIONAL_CA_KEY, ADDITIONAL_CA_KEY)) + ); + } + #[test] fn bootstrap_secrets_are_immutable_and_split_by_trust_domain() { let names = SandboxRuntimeNames::new("pair"); @@ -1442,6 +1570,7 @@ mod tests { false, staged, None, + None, owner(), &[], ) diff --git a/crates/openshell-driver-podman/Cargo.toml b/crates/openshell-driver-podman/Cargo.toml index b6c755e3f..bdbd59e3a 100644 --- a/crates/openshell-driver-podman/Cargo.toml +++ b/crates/openshell-driver-podman/Cargo.toml @@ -20,6 +20,7 @@ openshell-otel = { path = "../openshell-otel" } openshell-isolation-interface = { path = "../openshell-isolation-interface" } openshell-sandbox-backend = { path = "../openshell-sandbox-backend" } tar = "0.4" +tempfile = "3" uuid = { workspace = true } tokio = { workspace = true } @@ -44,6 +45,7 @@ tower-http = { workspace = true } http = { workspace = true } thiserror = { workspace = true } miette = { workspace = true } +sha2 = { workspace = true } url = { workspace = true } [dev-dependencies] @@ -53,7 +55,6 @@ opentelemetry_sdk = { workspace = true, features = ["testing"] } prost-types = { workspace = true } rcgen = { workspace = true } temp-env = "0.3" -tempfile = "3" tokio = { workspace = true, features = ["test-util"] } [lints] diff --git a/crates/openshell-driver-podman/src/client.rs b/crates/openshell-driver-podman/src/client.rs index 059b59402..92952dac2 100644 --- a/crates/openshell-driver-podman/src/client.rs +++ b/crates/openshell-driver-podman/src/client.rs @@ -47,7 +47,7 @@ pub enum PodmanApiError { /// Maximum resource name length. Podman container names become directory /// names in the storage driver, so we cap at 255 to stay within ext4/xfs /// filename limits. -const MAX_NAME_LEN: usize = 255; +pub const MAX_NAME_LEN: usize = 255; /// Validate that a resource name is safe for URL path interpolation. /// @@ -172,6 +172,12 @@ pub struct ContainerConfig { pub labels: HashMap, #[serde(default)] pub user: String, + /// The immutable image ID the container was created from. Podman records + /// the reference used at creation time here; workloads are always created + /// from a resolved image ID, so this pins the image the container runs + /// independently of any later tag movement. + #[serde(default)] + pub image: String, } /// Immutable image metadata needed to bind OCI identity inspection to launch. @@ -662,6 +668,40 @@ impl PodmanClient { } } + /// Rename a stopped container by name or ID. + /// + /// The libpod rename endpoint frees the old canonical name without + /// deleting the container. Reconciliation uses it as a rollback point + /// while a replacement is created. + pub async fn rename_container(&self, name: &str, new_name: &str) -> Result<(), PodmanApiError> { + validate_name(name)?; + validate_name(new_name)?; + self.request_ok( + hyper::Method::POST, + &format!("/libpod/containers/{name}/rename?name={new_name}"), + None, + ) + .await + } + + /// Remove a stopped container without asking Podman to remove volumes. + /// + /// This is deliberately separate from [`Self::remove_container`]: stopped + /// sandbox trust reconciliation replaces only the container and must never + /// remove its pre-existing named workspace volume. + pub async fn remove_container_preserving_volumes( + &self, + name: &str, + ) -> Result<(), PodmanApiError> { + validate_name(name)?; + self.request_ok( + hyper::Method::DELETE, + &format!("/libpod/containers/{name}?force=true&volumes=false&timeout=0"), + None, + ) + .await + } + /// Download a file from a container as a tar archive. /// /// Calls `GET /libpod/containers/{name}/archive?path={path}` and returns @@ -873,6 +913,30 @@ impl PodmanClient { } } + /// Check whether a named Podman secret exists without reading its value. + /// + /// A 404 is an expected compatibility signal for legacy tokenless + /// sandboxes, so return it as `Ok(false)` without parsing or exposing the + /// response body. All other API failures remain errors. + pub async fn secret_exists(&self, name: &str) -> Result { + validate_name(name)?; + let (status, bytes) = self + .request( + hyper::Method::GET, + &format!("/libpod/secrets/{name}/json"), + None, + API_TIMEOUT, + ) + .await?; + if status.is_success() { + Ok(true) + } else if status == hyper::StatusCode::NOT_FOUND { + Ok(false) + } else { + Err(error_from_response(status.as_u16(), &bytes)) + } + } + /// Remove a Podman secret by name. Idempotent (not-found is ignored). pub async fn remove_secret(&self, name: &str) -> Result<(), PodmanApiError> { validate_name(name)?; @@ -1341,6 +1405,58 @@ mod tests { let _ = std::fs::remove_file(socket_path); } + #[tokio::test] + async fn secret_exists_treats_not_found_as_absent_without_exposing_body() { + let (socket_path, request_log, handle) = spawn_podman_stub( + "secret-not-found", + vec![StubResponse::new( + StatusCode::NOT_FOUND, + "sensitive daemon diagnostic", + )], + ); + let client = PodmanClient::new(socket_path.clone()); + + assert!( + !client + .secret_exists("openshell-token-sandbox-1") + .await + .expect("404 must be a normal absent-secret result") + ); + handle.await.expect("stub should finish"); + assert_eq!( + request_log + .lock() + .expect("request log lock should not be poisoned") + .as_slice(), + ["GET /v5.0.0/libpod/secrets/openshell-token-sandbox-1/json"] + ); + let _ = std::fs::remove_file(socket_path); + } + + #[tokio::test] + async fn rename_container_uses_libpod_rename_endpoint() { + let (socket_path, request_log, handle) = spawn_podman_stub( + "rename-container", + vec![StubResponse::new(StatusCode::NO_CONTENT, "")], + ); + let client = PodmanClient::new(socket_path.clone()); + + client + .rename_container("sandbox-123", "sandbox-123-rollback-1") + .await + .expect("container rename should succeed"); + + handle.await.expect("stub task should finish"); + assert_eq!( + request_log + .lock() + .expect("request log lock should not be poisoned") + .as_slice(), + ["POST /v5.0.0/libpod/containers/sandbox-123/rename?name=sandbox-123-rollback-1"] + ); + let _ = std::fs::remove_file(socket_path); + } + #[tokio::test] async fn remove_container_uses_single_timed_libpod_removal() { let (socket_path, request_log, handle) = spawn_podman_stub( diff --git a/crates/openshell-driver-podman/src/config.rs b/crates/openshell-driver-podman/src/config.rs index 2aa8b9c0d..97177e97e 100644 --- a/crates/openshell-driver-podman/src/config.rs +++ b/crates/openshell-driver-podman/src/config.rs @@ -203,6 +203,15 @@ pub struct PodmanComputeConfig { /// system CA bundle. The bundle must exist and contain at least one /// certificate; this is validated on the gateway host at sandbox-create /// time, not deferred until the supervisor container starts. + /// + /// Populated only from the `--sandbox-additional-ca-bundle` CLI argument + /// (or `OPENSHELL_SANDBOX_ADDITIONAL_CA_BUNDLE`) for standalone drivers. + /// Deliberately excluded from driver JSON config (`#[serde(skip)]` + + /// `deny_unknown_fields`): gateway-managed deployments must use the + /// global `[openshell.supervisor.network]` setting so every driver + /// receives the same staged bundle, and a per-driver path in operator + /// JSON is rejected instead of silently diverging from it. + #[serde(skip)] pub additional_ca_bundle: Option, /// User namespace mode for sandbox containers (e.g. `auto`, `private`). /// When unset, containers use the default user namespace. @@ -649,6 +658,15 @@ mod tests { assert!(error.to_string().contains("sandbox_ssh_socket_path")); } + #[test] + fn config_rejects_driver_specific_destination_ca_setting() { + let error = serde_json::from_value::(serde_json::json!({ + "additional_ca_bundle": ["/operator/ca.crt"] + })) + .expect_err("destination trust is global, not a Podman driver setting"); + assert!(error.to_string().contains("additional_ca_bundle")); + } + #[test] fn default_config_disables_health_checks() { assert_eq!( diff --git a/crates/openshell-driver-podman/src/container.rs b/crates/openshell-driver-podman/src/container.rs index 8e7a9f422..59eaa2cf3 100644 --- a/crates/openshell-driver-podman/src/container.rs +++ b/crates/openshell-driver-podman/src/container.rs @@ -10,9 +10,12 @@ use openshell_core::driver_mounts::SelinuxLabel; use openshell_core::gpu::validate_cdi_device_names; #[cfg(test)] use openshell_core::gpu::{driver_gpu_requirements, validate_cdi_gpu_device_request}; +use openshell_core::network_trust::{ + NETWORK_SUPERVISOR_TRUST_GENERATION_KEY, NETWORK_SUPERVISOR_TRUST_GENERATION_NONE, +}; use openshell_core::proto::compute::v1::{DriverSandbox, DriverSandboxTemplate}; use openshell_core::proto_struct::deserialize_optional_non_empty_string_list; -use openshell_core::{driver_mounts, proto_struct}; +use openshell_core::{NetworkSupervisorTrustBundle, driver_mounts, proto_struct}; use serde::Serialize; use serde_json::Value; use std::collections::{BTreeMap, HashSet}; @@ -613,17 +616,51 @@ fn upstream_proxy_cli_args(config: &PodmanComputeConfig) -> Vec { args } -/// Build the additional-destination-CA command-line argument passed to the -/// supervisor. Independent of `upstream_proxy_cli_args`: this applies to -/// direct, inspected egress regardless of whether a corporate proxy is -/// configured. -fn additional_ca_cli_args(config: &PodmanComputeConfig) -> Vec { - let mut args = Vec::new(); - if config.additional_ca_bundle.is_some() { - args.push("--additional-ca-bundle".to_string()); - args.push(ADDITIONAL_CA_MOUNT_PATH.to_string()); +/// Build the destination-trust arguments for the trusted supervisor only. +/// +/// The source path stays out of argv; the digest pairs the mounted artifact +/// with the immutable generation recorded on the workload label. +fn network_trust_cli_args(network_trust: Option<&NetworkSupervisorTrustBundle>) -> Vec { + network_trust.map_or_else(Vec::new, |_bundle| { + vec![ + "--additional-ca-bundle".to_string(), + ADDITIONAL_CA_MOUNT_PATH.to_string(), + ] + }) +} + +fn network_trust_mount(bundle: &NetworkSupervisorTrustBundle) -> Result { + bundle.verify_artifact().map_err(|error| { + ComputeDriverError::Precondition(format!( + "verify gateway-owned network additional CA artifact: {error}" + )) + })?; + let path = bundle.artifact_path(); + let source = path.to_str().ok_or_else(|| { + ComputeDriverError::Precondition(format!( + "network additional CA artifact path is not valid UTF-8: {}", + path.display() + )) + })?; + driver_mounts::validate_absolute_mount_source(source, "network additional CA artifact") + .map_err(ComputeDriverError::Precondition)?; + + let mut options = vec!["ro".into(), "rbind".into()]; + if is_selinux_enabled() { + options.push("z".into()); } - args + Ok(Mount { + kind: "bind".into(), + source: source.to_string(), + destination: ADDITIONAL_CA_MOUNT_PATH.into(), + options, + }) +} + +fn network_trust_generation(network_trust: Option<&NetworkSupervisorTrustBundle>) -> &str { + network_trust.map_or(NETWORK_SUPERVISOR_TRUST_GENERATION_NONE, |bundle| { + bundle.digest() + }) } fn build_env( @@ -1300,7 +1337,6 @@ fn build_base_spec( image_volumes.extend(user_mounts.image_volumes); let mut command = vec!["--workdir".to_string(), image.workspace_root.clone()]; command.extend(upstream_proxy_cli_args(config)); - command.extend(additional_ca_cli_args(config)); let container_spec = ContainerSpec { name, @@ -1467,24 +1503,6 @@ fn build_base_spec( options: ro, }); } - // Bind-mount an operator-supplied additional destination CA - // bundle read-only when configured. Independent of the corporate - // proxy CA above: this applies to direct, inspected egress. The - // supervisor reads it via the --additional-ca-bundle argv path - // (see additional_ca_cli_args) and folds it into its upstream - // trust store alongside the system bundle. - if let Some(ca_bundle) = &config.additional_ca_bundle { - let mut ro = vec!["ro".into(), "rbind".into()]; - if is_selinux_enabled() { - ro.push("z".into()); - } - m.push(Mount { - kind: "bind".into(), - source: ca_bundle.clone(), - destination: ADDITIONAL_CA_MOUNT_PATH.into(), - options: ro, - }); - } if let Some(bin_path) = supervisor_bin_path { let mut opts = vec!["ro".into(), "rbind".into()]; if is_selinux_enabled() { @@ -1564,6 +1582,9 @@ pub struct IsolationSpecInput<'a> { pub image: &'a ResolvedPodmanImage, pub supervisor_bin: Option<&'a Path>, pub tls_secrets: Option<&'a [String; 1]>, + /// Gateway-owned immutable destination-trust artifact. It is deliberately + /// mounted only into the trusted supervisor spec. + pub network_trust: Option<&'a NetworkSupervisorTrustBundle>, pub identity: &'a openshell_isolation_interface::contract::ResolvedWorkloadIdentity, } @@ -1687,6 +1708,14 @@ pub fn build_isolation_specs( supervisor .labels .insert(crate::isolation::LABEL_ROLE.into(), "supervisor".into()); + // The workload is never replaced when the destination-trust bundle + // changes; only this supervisor is. Record the generation it was created + // with so a later start can detect a stale supervisor and recreate it + // without touching the workload or its data. + supervisor.labels.insert( + NETWORK_SUPERVISOR_TRUST_GENERATION_KEY.into(), + network_trust_generation(input.network_trust).into(), + ); supervisor.image.clone_from(&input.config.supervisor_image); supervisor.entrypoint = vec!["/openshell-supervisor".into()]; supervisor.command.extend([ @@ -1697,6 +1726,12 @@ pub fn build_isolation_specs( format!("--auth-bundle-file={}", crate::isolation::AUTH_BUNDLE_PATH), "--health-socket-path=/run/openshell/supervisor-health.sock".into(), ]); + supervisor + .command + .extend(network_trust_cli_args(input.network_trust)); + if let Some(network_trust) = input.network_trust { + supervisor.mounts.push(network_trust_mount(network_trust)?); + } supervisor.env.insert( openshell_core::sandbox_env::ADMITTED_ISOLATION_BACKEND.into(), openshell_sandbox_backend::BACKEND_NAME.into(), @@ -1919,6 +1954,122 @@ mod tests { .expect("fixture image metadata should be valid") } + #[test] + fn additional_destination_trust_is_delivered_only_to_supervisor_with_digest() { + let artifact = crate::test_utils::unique_socket_path("additional-ca").with_extension("crt"); + let pem = b"normalized additional CA fixture"; + std::fs::write(&artifact, pem).unwrap(); + #[cfg(unix)] + std::fs::set_permissions( + &artifact, + std::os::unix::fs::PermissionsExt::from_mode(0o444), + ) + .unwrap(); + let bundle = NetworkSupervisorTrustBundle::new( + pem.to_vec(), + 1, + "sha256:paired-digest", + artifact.clone(), + ); + let sandbox = DriverSandbox { + id: "trust-pair".into(), + name: "trust-pair".into(), + ..Default::default() + }; + let identity = openshell_isolation_interface::contract::ResolvedWorkloadIdentity::new( + 1000, + 1001, + Vec::new(), + "image".into(), + "sha256:image".into(), + ) + .unwrap(); + let image = resolved_image("image", "sha256:image", "1000:1001", "/workspace/project"); + let specs = build_isolation_specs(IsolationSpecInput { + sandbox: &sandbox, + config: &PodmanComputeConfig::default(), + token_secret: None, + resolver_secret: "resolver", + gpu_devices: None, + image: &image, + supervisor_bin: None, + tls_secrets: None, + network_trust: Some(&bundle), + identity: &identity, + }) + .unwrap(); + + assert_eq!( + specs + .supervisor + .labels + .get(NETWORK_SUPERVISOR_TRUST_GENERATION_KEY) + .map(String::as_str), + Some("sha256:paired-digest") + ); + assert!( + specs + .workload + .command + .iter() + .all(|argument| argument != "--additional-ca-bundle") + ); + assert!( + specs + .workload + .mounts + .iter() + .all(|mount| { mount.destination != ADDITIONAL_CA_MOUNT_PATH }) + ); + assert!(specs.supervisor.command.windows(2).any(|arguments| { + arguments == ["--additional-ca-bundle", ADDITIONAL_CA_MOUNT_PATH] + })); + assert!(specs.supervisor.mounts.iter().any(|mount| { + mount.destination == ADDITIONAL_CA_MOUNT_PATH + && mount.source == artifact.to_string_lossy() + && mount.options.iter().any(|option| option == "ro") + })); + let _ = std::fs::remove_file(artifact); + } + + #[test] + fn tampered_destination_trust_artifact_fails_closed_without_pem_contents() { + let artifact = + crate::test_utils::unique_socket_path("tampered-additional-ca").with_extension("crt"); + std::fs::write(&artifact, b"expected").unwrap(); + #[cfg(unix)] + std::fs::set_permissions( + &artifact, + std::os::unix::fs::PermissionsExt::from_mode(0o444), + ) + .unwrap(); + let bundle = NetworkSupervisorTrustBundle::new( + b"expected".to_vec(), + 1, + "sha256:expected", + artifact.clone(), + ); + std::fs::remove_file(&artifact).unwrap(); + std::fs::write(&artifact, b"BEGIN CERTIFICATE tampered contents").unwrap(); + #[cfg(unix)] + std::fs::set_permissions( + &artifact, + std::os::unix::fs::PermissionsExt::from_mode(0o444), + ) + .unwrap(); + + let Err(error) = network_trust_mount(&bundle) else { + panic!("tampering must fail closed"); + }; + assert!( + error + .to_string() + .contains("gateway-owned network additional CA artifact") + ); + assert!(!error.to_string().contains("tampered contents")); + let _ = std::fs::remove_file(artifact); + } + #[test] fn isolated_pair_keeps_privileges_network_and_secrets_out_of_workload() { let sandbox = DriverSandbox { @@ -1953,6 +2104,7 @@ mod tests { image: &image, supervisor_bin: None, tls_secrets: None, + network_trust: None, identity: &identity, }) .unwrap(); @@ -1974,6 +2126,14 @@ mod tests { assert_eq!(specs.supervisor.groups, vec!["2000"]); assert!(specs.supervisor.cap_add.is_empty()); assert_eq!(specs.workload.netns.nsmode, "none"); + assert_eq!( + specs + .supervisor + .labels + .get(NETWORK_SUPERVISOR_TRUST_GENERATION_KEY) + .map(String::as_str), + Some(NETWORK_SUPERVISOR_TRUST_GENERATION_NONE) + ); assert_eq!( specs .workload @@ -2006,6 +2166,7 @@ mod tests { image: &ResolvedPodmanImage::new("sha256:image"), supervisor_bin: None, tls_secrets: None, + network_trust: None, identity: &default_identity, }) .unwrap(); @@ -2961,127 +3122,17 @@ mod tests { ), "no CA bundle mount without operator config" ); - } - - #[test] - fn container_spec_binds_additional_ca_bundle_and_passes_argv_without_proxy() { - // #3781: the additional CA must work for direct, inspected egress - // with no corporate proxy configured at all — unlike proxy_ca_bundle, - // which requires https_proxy. - let sandbox = test_sandbox("additional-ca-id", "additional-ca-name"); - let mut config = test_config(); assert!( - config.https_proxy.is_none(), - "test must exercise the no-proxy case" - ); - config.additional_ca_bundle = Some("/host/additional-ca.pem".to_string()); - - let spec = build_container_spec(&sandbox, &config); - let command = spec_command(&spec); - - let idx = command - .iter() - .position(|a| a == "--additional-ca-bundle") - .expect("additional CA bundle flag present"); - assert_eq!( - command.get(idx + 1).map(String::as_str), - Some("/etc/openshell/tls/additional/ca-bundle.pem") - ); - - let mounts = spec["mounts"] - .as_array() - .expect("mounts should be an array"); - let ca_mount = mounts - .iter() - .find(|m| { - m["type"].as_str() == Some("bind") - && m["destination"].as_str() - == Some("/etc/openshell/tls/additional/ca-bundle.pem") - }) - .expect("additional CA bundle bind mount present"); - assert_eq!(ca_mount["source"].as_str(), Some("/host/additional-ca.pem")); - assert!( - ca_mount["options"] - .as_array() - .expect("options array") + !mounts .iter() - .any(|o| o.as_str() == Some("ro")), - "additional CA bundle mount must be read-only" - ); - } - - #[test] - fn isolated_pair_mounts_additional_ca_bundle_only_on_supervisor() { - // build_isolation_specs() filters the base spec's mounts per role, so - // the base-spec test above does not prove the final supervisor keeps - // the CA the --additional-ca-bundle argv points at. - let sandbox = test_sandbox("additional-ca-pair", "additional-ca-pair"); - let mut config = test_config(); - config.additional_ca_bundle = Some("/host/additional-ca.pem".to_string()); - let identity = openshell_isolation_interface::contract::ResolvedWorkloadIdentity::new( - 1000, - 1000, - Vec::new(), - "default".into(), - "sha256:image".into(), - ) - .unwrap(); - let specs = build_isolation_specs(IsolationSpecInput { - sandbox: &sandbox, - config: &config, - token_secret: Some("jwt"), - resolver_secret: "resolver", - gpu_devices: None, - image: &ResolvedPodmanImage::new("sha256:image"), - supervisor_bin: None, - tls_secrets: None, - identity: &identity, - }) - .unwrap(); - - let ca_mount = specs - .supervisor - .mounts - .iter() - .find(|mount| mount.destination == ADDITIONAL_CA_MOUNT_PATH) - .expect("supervisor keeps the additional CA bind mount"); - assert_eq!(ca_mount.kind, "bind"); - assert_eq!(ca_mount.source, "/host/additional-ca.pem"); - assert!(ca_mount.options.contains(&"ro".to_string())); - let idx = specs - .supervisor - .command - .iter() - .position(|arg| arg == "--additional-ca-bundle") - .expect("supervisor argv carries the additional CA flag"); - assert_eq!( - specs.supervisor.command.get(idx + 1).map(String::as_str), - Some(ADDITIONAL_CA_MOUNT_PATH) + .any(|m| { m["destination"].as_str() == Some(ADDITIONAL_CA_MOUNT_PATH) }), + "no destination CA mount without global config" ); assert!( - specs - .workload - .mounts + !spec_command(&spec) .iter() - .all(|mount| mount.destination != ADDITIONAL_CA_MOUNT_PATH), - "workload must not receive the additional CA mount" - ); - } - - #[test] - fn container_spec_omits_additional_ca_bundle_when_unconfigured() { - let sandbox = test_sandbox("test-id", "test-name"); - let spec = build_container_spec(&sandbox, &test_config()); - let command = spec_command(&spec); - assert!( - !command.iter().any(|a| a == "--additional-ca-bundle"), - "no additional-ca-bundle flag without operator config" - ); - let mounts = spec["mounts"].as_array().expect("mounts array"); - assert!( - !mounts.iter().any(|m| m["destination"].as_str() - == Some("/etc/openshell/tls/additional/ca-bundle.pem")), - "no additional CA bundle mount without operator config" + .any(|arg| arg == "--additional-ca-bundle"), + "no destination CA argument without global config" ); } @@ -3766,25 +3817,30 @@ mod tests { fn driver_config_rejects_reserved_mount_targets() { use openshell_core::proto::compute::v1::{DriverSandboxSpec, DriverSandboxTemplate}; - let mut sandbox = test_sandbox("test-id", "test-name"); - sandbox.spec = Some(DriverSandboxSpec { - template: Some(DriverSandboxTemplate { - driver_config: Some(json_struct(serde_json::json!({ - "mounts": [{ - "type": "volume", - "source": "work-nfs", - "target": "/etc/openshell/tls/client" - }] - }))), + for target in ["/etc/openshell/tls/client", ADDITIONAL_CA_MOUNT_PATH] { + let mut sandbox = test_sandbox("test-id", "test-name"); + sandbox.spec = Some(DriverSandboxSpec { + template: Some(DriverSandboxTemplate { + driver_config: Some(json_struct(serde_json::json!({ + "mounts": [{ + "type": "volume", + "source": "work-nfs", + "target": target + }] + }))), + ..Default::default() + }), ..Default::default() - }), - ..Default::default() - }); - let config = test_config(); + }); + let config = test_config(); - let err = try_build_container_spec_with_token(&sandbox, &config, None).unwrap_err(); + let err = try_build_container_spec_with_token(&sandbox, &config, None).unwrap_err(); - assert!(err.to_string().contains("reserved OpenShell path")); + assert!( + err.to_string().contains("reserved OpenShell path"), + "target {target}: {err}" + ); + } } #[test] diff --git a/crates/openshell-driver-podman/src/driver.rs b/crates/openshell-driver-podman/src/driver.rs index 06d4d7b6f..4a9a812a5 100644 --- a/crates/openshell-driver-podman/src/driver.rs +++ b/crates/openshell-driver-podman/src/driver.rs @@ -10,7 +10,6 @@ use crate::watcher::{ self, LifecycleEventFences, WatchStream, driver_sandbox_from_inspect, driver_sandbox_from_list_entry, }; -use openshell_core::ComputeDriverError; use openshell_core::config::CDI_GPU_DEVICE_ALL; use openshell_core::driver_utils::{ SANDBOX_RUNTIME_IMAGE_BINARY_PATH, extract_first_tar_entry, supervisor_image_should_refresh, @@ -20,14 +19,20 @@ use openshell_core::gpu::{ CdiGpuDefaultSelector, CdiGpuInventory, CdiGpuSelectionError, driver_gpu_requirements, effective_driver_gpu_count, validate_cdi_gpu_device_request, }; +use openshell_core::network_trust::{ + NETWORK_SUPERVISOR_TRUST_GENERATION_KEY, NETWORK_SUPERVISOR_TRUST_GENERATION_NONE, +}; use openshell_core::proto::compute::v1::{ CpuResourceCapabilities, DriverSandbox, GetCapabilitiesResponse, GpuResourceCapabilities, GpuResourceRequirements, MemoryResourceCapabilities, ResourceCapabilities, }; +use openshell_core::{ComputeDriverError, NetworkSupervisorTrustBundle}; use std::collections::HashMap; use std::path::{Path, PathBuf}; use std::sync::Arc; use std::time::Duration; + +use sha2::Digest; use tracing::{Instrument as _, debug, info, warn}; const STOP_COMPLETION_POLL_INTERVAL: Duration = Duration::from_millis(50); @@ -161,6 +166,16 @@ pub struct PodmanComputeDriver { gpu_selector: Arc, gpu_inventory_refresh: Arc (CdiGpuInventory, bool) + Send + Sync>, lifecycle_event_fences: LifecycleEventFences, + /// Gateway-normalized destination trust held outside untrusted sandbox + /// specs. Gateway-managed drivers receive the staged bundle from the + /// gateway; standalone drivers self-source one from the + /// `--sandbox-additional-ca-bundle` operator path. + network_trust: Option, + /// Keeps the driver-owned staging directory for an operator-supplied + /// additional CA bundle alive for the driver process lifetime. The + /// artifact inside it is bind-mounted into supervisors and re-verified + /// on every create/start. + network_trust_staging: Option>, } impl std::fmt::Debug for PodmanComputeDriver { @@ -170,6 +185,10 @@ impl std::fmt::Debug for PodmanComputeDriver { .field("default_image", &self.config.default_image) .field("network_name", &self.config.network_name) .field("gpu_inventory", &self.gpu_selector.device_ids()) + .field( + "network_trust_staged", + &self.network_trust_staging.is_some(), + ) .finish() } } @@ -460,15 +479,121 @@ fn resolve_socket_path( }) } +/// Build destination trust from an operator-supplied standalone bundle path. +/// +/// Standalone (non-gateway) drivers accept `--sandbox-additional-ca-bundle` +/// directly. The file is read with the same bounded, no-symlink validation +/// the gateway applies to its own configured bundle, and a SHA-256 digest of +/// the material becomes the trust generation so stop/start reconciliation +/// behaves identically to gateway-managed deployments. The operator-owned +/// source keeps its own permissions; the normalized material is staged into +/// a driver-owned `0444` artifact so the immutable-artifact verification +/// every later create/start performs accepts it. +fn operator_network_trust( + path: &str, +) -> Result<(NetworkSupervisorTrustBundle, tempfile::TempDir), PodmanApiError> { + let pem = openshell_core::driver_utils::read_upstream_proxy_ca_bundle_file( + path, + "additional_ca_bundle", + ) + .map_err(PodmanApiError::InvalidInput)?; + let certificate_count = pem.matches("-----BEGIN CERTIFICATE-----").count(); + // Deterministic destination-trust generation label, not an authenticity + // proof: trust comes from the bounded validation above, the immutable + // staged artifact, and TLS verification. Classified as a non-security + // identity hash in openshell-crypto/README.md. + let digest = format!("sha256:{:x}", sha2::Sha256::digest(pem.as_bytes())); + + let staging = tempfile::tempdir().map_err(|error| { + PodmanApiError::InvalidInput(format!("create additional CA staging directory: {error}")) + })?; + let artifact_path = staging.path().join("additional-ca.crt"); + stage_operator_network_trust_artifact(&artifact_path, pem.as_bytes())?; + Ok(( + NetworkSupervisorTrustBundle::new( + pem.into_bytes(), + certificate_count, + digest, + artifact_path, + ), + staging, + )) +} + +/// Write the normalized operator material as a read-only, regular artifact. +/// +/// The file is created writable-private and only then made immutable so no +/// observer ever sees a world-writable artifact; the staging directory itself +/// is process-private (`0700`). +fn stage_operator_network_trust_artifact( + artifact_path: &Path, + normalized: &[u8], +) -> Result<(), PodmanApiError> { + use std::io::Write as _; + let mut options = std::fs::OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(0o600); + } + let mut file = options.open(artifact_path).map_err(|error| { + PodmanApiError::InvalidInput(format!( + "create additional CA artifact '{}': {error}", + artifact_path.display() + )) + })?; + file.write_all(normalized) + .and_then(|()| file.sync_all()) + .map_err(|error| { + PodmanApiError::InvalidInput(format!( + "write additional CA artifact '{}': {error}", + artifact_path.display() + )) + })?; + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + std::fs::set_permissions(artifact_path, std::fs::Permissions::from_mode(0o444)).map_err( + |error| { + PodmanApiError::InvalidInput(format!( + "restrict additional CA artifact '{}': {error}", + artifact_path.display() + )) + }, + )?; + } + Ok(()) +} + impl PodmanComputeDriver { /// Create a new driver, verifying the Podman socket is reachable. - pub async fn new(mut config: PodmanComputeConfig) -> Result { + pub async fn new( + mut config: PodmanComputeConfig, + network_trust: Option, + ) -> Result { const MAX_PING_RETRIES: u32 = 5; const PING_RETRY_DELAY: Duration = Duration::from_secs(2); let socket_path = resolve_socket_path(config.socket_path.clone(), detect_socket)?; config.socket_path = Some(socket_path.clone()); + let (network_trust, network_trust_staging) = + match (network_trust, config.additional_ca_bundle.as_deref()) { + (Some(_), Some(path)) => { + return Err(PodmanApiError::InvalidInput(format!( + "destination CA trust was supplied both by the gateway and through \ + --sandbox-additional-ca-bundle ('{path}'); configure exactly one source" + ))); + } + (Some(bundle), None) => (Some(bundle), None), + (None, Some(path)) => { + let (bundle, staging) = operator_network_trust(path)?; + (Some(bundle), Some(Arc::new(staging))) + } + (None, None) => (None, None), + }; + if !socket_path.exists() { if cfg!(target_os = "macos") { warn!( @@ -589,6 +714,8 @@ impl PodmanComputeDriver { )), gpu_inventory_refresh: Arc::new(local_podman_gpu_selector_state), lifecycle_event_fences: LifecycleEventFences::default(), + network_trust, + network_trust_staging, }; let reconciler = driver.clone(); tokio::spawn(async move { @@ -925,6 +1052,24 @@ impl PodmanComputeDriver { } } + fn verify_network_trust(&self) -> Result<(), ComputeDriverError> { + if let Some(bundle) = self.network_trust.as_ref() { + bundle.verify_artifact().map_err(|error| { + ComputeDriverError::Precondition(format!( + "verify gateway-owned network additional CA artifact: {error}" + )) + })?; + } + Ok(()) + } + + fn expected_network_trust_generation(&self) -> &str { + self.network_trust.as_ref().map_or( + NETWORK_SUPERVISOR_TRUST_GENERATION_NONE, + NetworkSupervisorTrustBundle::digest, + ) + } + /// Create a sandbox container. #[tracing::instrument( name = "podman.provision", @@ -937,7 +1082,28 @@ impl PodmanComputeDriver { ) )] pub async fn create_sandbox(&self, sandbox: &DriverSandbox) -> Result<(), ComputeDriverError> { + self.create_sandbox_with_existing_token(sandbox, None, None, None) + .await + } + + /// Provision a replacement while retaining the existing token secret. + /// Only stopped-pair reconciliation uses this; ordinary creates always + /// generate their own secret from the gateway's create credential. + async fn create_sandbox_with_existing_token( + &self, + sandbox: &DriverSandbox, + existing_token_secret: Option<&str>, + restart_generation: Option<&str>, + existing_workload_container_id: Option<&str>, + ) -> Result<(), ComputeDriverError> { let span_status = openshell_otel::ErrorStatusGuard::current(); + let reuse_existing_resources = restart_generation.is_some(); + let restart_generation = + restart_generation.map_or_else(|| uuid::Uuid::new_v4().to_string(), ToOwned::to_owned); + // The same immutable gateway snapshot is verified immediately before + // any workload/supervisor pair is admitted. Do not turn a tampered + // artifact into a runtime-only supervisor failure. + self.verify_network_trust()?; if sandbox.name.is_empty() { return Err(ComputeDriverError::Precondition( "sandbox name is required".into(), @@ -1007,23 +1173,26 @@ impl PodmanComputeDriver { )); } let pull_policy = podman_image_pull_policy(self.config.image_pull_policy); - info!(image = %image, policy = %pull_policy, "Ensuring sandbox image"); - self.client - .pull_image(image, pull_policy) - .await - .map_err(ComputeDriverError::from)?; - let inspected_image = self - .client - .inspect_image(image) - .await - .map_err(ComputeDriverError::from)?; - if inspected_image.id.is_empty() { - return Err(ComputeDriverError::Precondition(format!( - "podman image '{image}' inspection did not return an immutable image ID" - ))); - } - let resolved_image = - container::ResolvedPodmanImage::from_inspect(image, inspected_image)?; + let resolved_image = if let Some(existing) = existing_workload_container_id { + self.resolve_retained_workload_image(existing, image).await? + } else { + info!(image = %image, policy = %pull_policy, "Ensuring sandbox image"); + self.client + .pull_image(image, pull_policy) + .await + .map_err(ComputeDriverError::from)?; + let inspected_image = self + .client + .inspect_image(image) + .await + .map_err(ComputeDriverError::from)?; + if inspected_image.id.is_empty() { + return Err(ComputeDriverError::Precondition(format!( + "podman image '{image}' inspection did not return an immutable image ID" + ))); + } + container::ResolvedPodmanImage::from_inspect(image, inspected_image)? + }; for mount_image in container::podman_driver_image_mount_sources( sandbox, @@ -1089,11 +1258,18 @@ impl PodmanComputeDriver { )); } - // Create the managed workspace volume, if needed, and runtime files. - let (resolver_secret_name, token_secret_name, proxy_auth_secret_name) = async { + // Create the workspace volume and per-sandbox runtime files. During a + // stopped trust replacement, retain the named resources belonging to + // the old pair and reuse its resolver/token secrets. + let ( + resolver_secret_name, + resolver_secret_owned, + token_secret_name, + proxy_auth_secret_name, + ) = async { let phase_status = openshell_otel::ErrorStatusGuard::current(); let result = async { - if managed_workspace { + if !reuse_existing_resources && managed_workspace { self.client .create_owned_volume( &vol_name, @@ -1104,22 +1280,44 @@ impl PodmanComputeDriver { .await .map_err(ComputeDriverError::from)?; } - let resolver_secret_name = - match create_sandbox_resolver_secret(&self.client, &sandbox.id).await { + let resolver_secret_name = container::resolver_secret_name(&sandbox.id); + let resolver_secret_owned = if reuse_existing_resources { + if !self + .client + .secret_exists(&resolver_secret_name) + .await + .map_err(ComputeDriverError::from)? + { + return Err(ComputeDriverError::Precondition( + "Podman trust replacement requires the existing resolver secret".into(), + )); + } + false + } else { + create_sandbox_resolver_secret(&self.client, &sandbox.id).await?; + true + }; + let token_secret_name = if let Some(secret) = existing_token_secret { + // The durable gateway Sandbox intentionally does not + // contain the bearer token. Reuse its per-sandbox + // secret after the old stopped workload is renamed. + Some(secret.to_string()) + } else { + match create_sandbox_token_secret(&self.client, sandbox).await { Ok(name) => name, Err(e) => { - let _ = self.client.remove_volume(&vol_name).await; + if !reuse_existing_resources { + let _ = self.client.remove_volume(&vol_name).await; + if resolver_secret_owned { + cleanup_sandbox_resolver_secret( + &self.client, + &resolver_secret_name, + ) + .await; + } + } return Err(e); } - }; - let token_secret_name = match create_sandbox_token_secret(&self.client, sandbox) - .await - { - Ok(name) => name, - Err(e) => { - let _ = self.client.remove_volume(&vol_name).await; - cleanup_sandbox_resolver_secret(&self.client, &resolver_secret_name).await; - return Err(e); } }; let proxy_auth_secret_name = @@ -1128,17 +1326,27 @@ impl PodmanComputeDriver { { Ok(name) => name, Err(e) => { - let _ = self.client.remove_volume(&vol_name).await; - if let Some(secret) = token_secret_name.as_deref() { + if !reuse_existing_resources { + let _ = self.client.remove_volume(&vol_name).await; + } + if existing_token_secret.is_none() + && let Some(secret) = token_secret_name.as_deref() + { cleanup_sandbox_token_secret(&self.client, secret).await; } - cleanup_sandbox_resolver_secret(&self.client, &resolver_secret_name) + if resolver_secret_owned { + cleanup_sandbox_resolver_secret( + &self.client, + &resolver_secret_name, + ) .await; + } return Err(e); } }; Ok(( resolver_secret_name, + resolver_secret_owned, token_secret_name, proxy_auth_secret_name, )) @@ -1155,16 +1363,24 @@ impl PodmanComputeDriver { .await?; // Clean up the volume and per-sandbox secrets on any failure past this - // point. + // point. Resources retained for a stopped trust replacement are left + // intact so the rollback path can restore the old pair. let channel_owned = std::sync::atomic::AtomicBool::new(false); let cleanup_created = || async { - if channel_owned.load(std::sync::atomic::Ordering::Relaxed) { + if !reuse_existing_resources && channel_owned.load(std::sync::atomic::Ordering::Relaxed) + { let _ = self.client.remove_volume(&channel_volume).await; let _ = self.client.remove_volume(&control_volume).await; } - let _ = self.client.remove_volume(&vol_name).await; - cleanup_sandbox_resolver_secret(&self.client, &resolver_secret_name).await; - if let Some(secret) = token_secret_name.as_deref() { + if !reuse_existing_resources { + let _ = self.client.remove_volume(&vol_name).await; + } + if resolver_secret_owned { + cleanup_sandbox_resolver_secret(&self.client, &resolver_secret_name).await; + } + if existing_token_secret.is_none() + && let Some(secret) = token_secret_name.as_deref() + { cleanup_sandbox_token_secret(&self.client, secret).await; } if let Some(secret) = proxy_auth_secret_name.as_deref() { @@ -1199,20 +1415,43 @@ impl PodmanComputeDriver { } }; - let tls_secret_names = if self.config.tls_enabled() { - let names = container::tls_secret_names(&sandbox.id); - if let Err(e) = create_tls_secrets(&self.client, &self.config, &names).await { - cleanup_created().await; - return Err(e); + let tls_secret_names = self + .config + .tls_enabled() + .then(|| container::tls_secret_names(&sandbox.id)); + let tls_secrets_owned = if let Some(names) = &tls_secret_names { + if reuse_existing_resources { + // The renamed stopped pair still owns the per-sandbox + // TLS secret under the same sandbox id, and the + // gateway CA it carries is unchanged; the replacement + // pair reuses it instead of colliding on the name. + if !self + .client + .secret_exists(&names[0]) + .await + .map_err(ComputeDriverError::from)? + { + return Err(ComputeDriverError::Precondition( + "Podman trust replacement requires the existing TLS CA secret" + .into(), + )); + } + false + } else { + if let Err(e) = create_tls_secrets(&self.client, &self.config, names).await + { + cleanup_created().await; + return Err(e); + } + true } - Some(names) } else { - None + false }; let cleanup_all = || async { cleanup_created().await; - if let Some(names) = &tls_secret_names { + if tls_secrets_owned && let Some(names) = &tls_secret_names { cleanup_tls_secrets(&self.client, names).await; } }; @@ -1226,6 +1465,7 @@ impl PodmanComputeDriver { image: &resolved_image, supervisor_bin: supervisor_bin_path.as_deref(), tls_secrets: tls_secret_names.as_ref(), + network_trust: self.network_trust.as_ref(), identity: &identity, }); let mut specs = match specs { @@ -1240,15 +1480,35 @@ impl PodmanComputeDriver { let create_result = async { let identities = self.validate_user_volume_mounts_available(sandbox).await?; specs.record_resource_identities(&identities)?; - self.client - .create_owned_volume(&channel_volume, &sandbox.id, &sandbox.workspace, None) - .await?; - channel_owned.store(true, std::sync::atomic::Ordering::Relaxed); - self.client - .create_owned_volume(&control_volume, &sandbox.id, &sandbox.workspace, None) - .await?; - let workload_id = self.client.create_typed_container(&specs.workload).await?; - created_workload = Some(workload_id.clone()); + if !reuse_existing_resources { + self.client + .create_owned_volume( + &channel_volume, + &sandbox.id, + &sandbox.workspace, + None, + ) + .await?; + channel_owned.store(true, std::sync::atomic::Ordering::Relaxed); + self.client + .create_owned_volume( + &control_volume, + &sandbox.id, + &sandbox.workspace, + None, + ) + .await?; + } + // Supervisor-only trust reconciliation reuses the stopped + // workload container together with its volumes, secrets, + // and workspace data. Only the supervisor is replaced. + let workload_id = if let Some(existing) = existing_workload_container_id { + existing.to_string() + } else { + let id = self.client.create_typed_container(&specs.workload).await?; + created_workload = Some(id.clone()); + id + }; self.client.verify_isolation_fence(&workload_id).await?; self.admit_container_resources(&workload_id).await?; let child_env = podman_child_environment(sandbox, &resolved_image.environment); @@ -1264,13 +1524,12 @@ impl PodmanComputeDriver { .and_then(|spec| { decode_launch_authentication(&spec.launch_authentication) })?; - let generation = uuid::Uuid::new_v4().to_string(); let archives = crate::isolation::bootstrap_archives( crate::isolation::BootstrapArchivesInput { runtime_initiated: true, sandbox_id: &sandbox.id, container_id: &workload_id, - generation: &generation, + generation: &restart_generation, host_gateway_ip, identity: &identity, allow_extra_supplementary_groups: @@ -1297,7 +1556,7 @@ impl PodmanComputeDriver { ) .await?; } - if managed_workspace { + if managed_workspace && existing_workload_container_id.is_none() { self.client .copy_to_container(&workload_id, "/sandbox", archives.workspace) .await?; @@ -1351,6 +1610,48 @@ impl PodmanComputeDriver { span_status.finish(Ok(())) } + /// Resolve the image identity for a supervisor-only refresh of a retained + /// workload container. + /// + /// The retained workload container keeps the image it was created from. + /// Re-resolving the request tag could select a different image whose user + /// and environment no longer match the retained workload, leaving + /// mode-0600 bootstrap files it cannot read. Pin to the immutable image ID + /// recorded on the stopped workload and never pull or resolve its tag + /// during refresh. + async fn resolve_retained_workload_image( + &self, + existing_workload_container_id: &str, + requested_image: &str, + ) -> Result { + let workload = self + .client + .inspect_container(existing_workload_container_id) + .await + .map_err(ComputeDriverError::from)?; + let pinned_image = workload.config.image; + if pinned_image.is_empty() { + return Err(ComputeDriverError::Precondition( + "retained Podman workload does not record its pinned image ID".into(), + )); + } + info!( + image = %pinned_image, + "Reusing retained workload image for supervisor-only refresh" + ); + let inspected_image = self + .client + .inspect_image(&pinned_image) + .await + .map_err(ComputeDriverError::from)?; + if inspected_image.id.is_empty() { + return Err(ComputeDriverError::Precondition(format!( + "podman image '{pinned_image}' inspection did not return an immutable image ID" + ))); + } + container::ResolvedPodmanImage::from_inspect(requested_image, inspected_image) + } + /// Resolve image accounts without executing any image-supplied program. async fn resolve_workload_identity( &self, @@ -1534,6 +1835,170 @@ impl PodmanComputeDriver { span_status.finish(result) } + /// Whether the trusted supervisor of a sandbox predates the gateway's + /// current destination-trust bundle. A missing supervisor or a missing + /// generation label (a supervisor created before this labeling existed) + /// counts as stale: only the supervisor is recreated to refresh it, never + /// the workload or its data. + async fn supervisor_network_trust_stale( + &self, + sandbox_id: &str, + ) -> Result { + let supervisor = crate::isolation::supervisor_name(sandbox_id); + match self.client.inspect_container(&supervisor).await { + Ok(inspect) => Ok(inspect + .config + .labels + .get(NETWORK_SUPERVISOR_TRUST_GENERATION_KEY) + .map(String::as_str) + != Some(self.expected_network_trust_generation())), + Err(PodmanApiError::NotFound(_)) => Ok(true), + Err(error) => Err(error.into()), + } + } + + /// Recreate only the trusted supervisor of a stopped pair so the start + /// picks up the gateway's current destination-trust bundle. The workload + /// container, its volumes, its secrets, and its workspace data are never + /// replaced: the additional CA bundle is mounted only into the supervisor. + async fn replace_supervisor_for_network_trust( + &self, + old_workload: &ContainerListEntry, + durable_sandbox: &DriverSandbox, + generation: &openshell_core::sandbox_generation::SandboxGenerationId, + encoded_authentication: &[u8], + ) -> Result<(), ComputeDriverError> { + if durable_sandbox.id + != old_workload + .labels + .get(LABEL_SANDBOX_ID) + .map_or("", String::as_str) + { + return Err(ComputeDriverError::Precondition( + "durable sandbox id did not match stopped Podman workload".into(), + )); + } + if durable_sandbox.name.is_empty() || durable_sandbox.spec.is_none() { + return Err(ComputeDriverError::Precondition( + "a durable sandbox provisioning snapshot is required for Podman supervisor trust reconciliation".into(), + )); + } + if encoded_authentication.is_empty() { + return Err(ComputeDriverError::Precondition( + "fresh launch authentication is required for Podman supervisor trust reconciliation" + .into(), + )); + } + // Validate all untrusted fields before changing the old supervisor. + // Image selection below is driven exclusively by this durable + // snapshot. + self.validated_sandbox_create(durable_sandbox).await?; + + let canonical_supervisor = crate::isolation::supervisor_name(&durable_sandbox.id); + let previous_supervisor = match self.client.inspect_container(&canonical_supervisor).await { + Ok(inspect) if inspect.state.running => { + return Err(ComputeDriverError::Precondition( + "refusing to replace a running Podman supervisor during trust reconciliation" + .into(), + )); + } + Ok(inspect) => Some(inspect.id), + Err(PodmanApiError::NotFound(_)) => None, + Err(error) => return Err(error.into()), + }; + + // Complete every remaining fallible preparation before the rename so + // a transient failure cannot return with the previous supervisor + // already moved to its rollback name, where a normal delete would + // never find it. + let existing_token_name = container::token_secret_name(&durable_sandbox.id); + let existing_token = self + .client + .secret_exists(&existing_token_name) + .await + .map_err(ComputeDriverError::from)? + .then_some(existing_token_name.as_str()); + let mut replacement = durable_sandbox.clone(); + replacement + .spec + .as_mut() + .expect("validated durable snapshot has a spec") + .launch_authentication = encoded_authentication.to_vec(); + + let supervisor_rollback_name = + format!("openshell-supervisor-rollback-{}", uuid::Uuid::new_v4()); + if previous_supervisor.is_some() + && let Err(error) = self + .client + .rename_container(&canonical_supervisor, &supervisor_rollback_name) + .await + { + return Err(error.into()); + } + + let provisioned = self + .create_sandbox_with_existing_token( + &replacement, + existing_token, + Some(generation.as_str()), + Some(&old_workload.id), + ) + .await; + if let Err(error) = provisioned { + // The stopped workload and all of its resources were retained. + // Restore the canonical supervisor name so a failed replacement + // has no externally observable provisioning change. + if previous_supervisor.is_some() + && let Err(restore_error) = self + .client + .rename_container(&supervisor_rollback_name, &canonical_supervisor) + .await + { + // The previous supervisor keeps the rollback name and must + // stay identifiable for operator cleanup. A later retry + // still succeeds: the canonical name is absent, so the + // replacement path recreates a fresh supervisor. + warn!( + rollback_container = %supervisor_rollback_name, + sandbox_id = %durable_sandbox.id, + restore_error = %restore_error, + "failed to restore the previous Podman supervisor after a failed trust refresh" + ); + return Err(ComputeDriverError::Message(format!( + "trust refresh failed and the previous supervisor could not be \ + restored; the retained supervisor is named '{supervisor_rollback_name}' \ + and can be removed manually: {restore_error}" + ))); + } + return Err(error); + } + + if previous_supervisor.is_some() + && let Err(error) = self + .client + .remove_container_preserving_volumes(&supervisor_rollback_name) + .await + && !matches!(error, PodmanApiError::NotFound(_)) + { + // The replacement supervisor is running; the stopped rollback + // container is an identifiable orphan. Surface the failure so + // the orphan is not silently lost; a retry observes the fresh + // canonical supervisor and resumes the normal start path. + warn!( + rollback_container = %supervisor_rollback_name, + sandbox_id = %durable_sandbox.id, + remove_error = %error, + "failed to remove the previous Podman supervisor after a trust refresh" + ); + return Err(ComputeDriverError::Message(format!( + "trust refresh succeeded but the previous supervisor could not be \ + removed; the orphaned supervisor is named '{supervisor_rollback_name}' \ + and can be removed manually: {error}" + ))); + } + Ok(()) + } + /// Start a previously stopped sandbox container. #[tracing::instrument( name = "podman.start_sandbox", @@ -1549,19 +2014,27 @@ impl PodmanComputeDriver { sandbox_id: &str, generation_id: &str, encoded_authentication: &[u8], + durable_sandbox: Option<&DriverSandbox>, ) -> Result<(), ComputeDriverError> { let span_status = openshell_otel::ErrorStatusGuard::current(); + // Verify before inspecting or modifying either member of the pair. + // The durable snapshot is used only if a stopped generation must be + // reconciled; it never comes from Podman's mutable inspect response. + self.verify_network_trust()?; let generation = openshell_core::sandbox_generation::SandboxGenerationId::parse( generation_id.to_string(), ) .map_err(|error| ComputeDriverError::InvalidArgument(error.to_string()))?; - let launch_authentication = decode_launch_authentication(encoded_authentication)?; + let launch_authentication = (!encoded_authentication.is_empty()) + .then(|| decode_launch_authentication(encoded_authentication)) + .transpose()?; let container = self .find_container(sandbox_id) .await? .ok_or(ComputeDriverError::NotFound)?; self.admit_container_resources(&container.id).await?; - if container.state == "running" { + let workload_running = container.state == "running"; + if workload_running { let supervisor = self .client .inspect_container(&crate::isolation::supervisor_name(sandbox_id)) @@ -1597,6 +2070,29 @@ impl PodmanComputeDriver { self.wait_for_container_stopped(sandbox_id, &container.id) .await?; } + // A user-driven start of a stopped sandbox mints fresh launch + // credentials. When the trusted supervisor predates the gateway's + // current destination-trust bundle (including supervisors created + // before generation labels existed), recreate only the supervisor so + // stop/start picks up the current additional CA bundle. The workload + // container, its volumes, and its workspace data are never replaced. + // Recovery without fresh credentials and running sandboxes keep their + // existing supervisor and bundle unchanged. + if !workload_running + && launch_authentication.is_some() + && let Some(durable_sandbox) = durable_sandbox + && self.supervisor_network_trust_stale(sandbox_id).await? + { + let result = self + .replace_supervisor_for_network_trust( + &container, + durable_sandbox, + &generation, + encoded_authentication, + ) + .await; + return span_status.finish(result); + } let container_id = container.id; info!(sandbox_id = %sandbox_id, container = %container_id, "Starting sandbox container"); @@ -1626,6 +2122,25 @@ impl PodmanComputeDriver { let bundle = extract_first_tar_entry(&archive).map_err(ComputeDriverError::Precondition)?; let restart_metadata = crate::isolation::restart_metadata_from_slice(&bundle)?; + if launch_authentication.is_none() { + if restart_metadata.generation != generation.as_str() { + return Err(ComputeDriverError::Precondition(format!( + "Podman sandbox is already provisioned for generation {}", + restart_metadata.generation + ))); + } + // A gateway crash can recover a persisted Starting transition + // without a newly minted credential. Its stopped pair already + // contains the authenticated channel and supervisor state, so + // restart it as-is rather than fabricating provisioning input. + self.client.verify_isolation_fence(&container_id).await?; + self.client.start_container(&container_id).await?; + if let Err(error) = self.client.start_container(&supervisor).await { + let _ = self.client.stop_container(&container_id, 0).await; + return Err(error.into()); + } + return Ok(()); + } let archives = crate::isolation::bootstrap_archives(crate::isolation::BootstrapArchivesInput { runtime_initiated: restart_metadata.runtime_initiated, @@ -1642,7 +2157,7 @@ impl PodmanComputeDriver { self.config.userns.as_deref(), ), child_env: restart_metadata.child_env, - launch_authentication: &launch_authentication, + launch_authentication: launch_authentication.as_ref().expect("checked above"), })?; self.client .copy_to_container( @@ -1913,6 +2428,8 @@ impl PodmanComputeDriver { (refresh_inventory.clone(), allow_all_default_gpu) }), lifecycle_event_fences: LifecycleEventFences::default(), + network_trust: None, + network_trust_staging: None, } } } @@ -2240,6 +2757,200 @@ mod tests { assert!(matches!(err, ComputeDriverError::AlreadyExists)); } + #[tokio::test] + async fn supervisor_refresh_pins_retained_workload_image_without_pulling_tag() { + let (socket, requests, handle) = spawn_podman_stub( + "refresh-pinned-image", + vec![ + // Inspect of the retained, stopped workload container: it was + // created from an immutable image ID. + StubResponse::new( + StatusCode::OK, + r#"{"Id":"ctr-retained","Name":"workload","State":{"Status":"exited","Running":false},"Config":{"Image":"sha256:pinned-image-id"}}"#, + ), + // Inspection of the pinned image by ID, not by its tag. + StubResponse::new( + StatusCode::OK, + r#"{"Id":"sha256:pinned-image-id","Config":{"User":"1000:1000","Env":["PATH=/usr/bin"],"WorkingDir":"/"}}"#, + ), + ], + ); + let driver = test_driver(socket); + let resolved = driver + .resolve_retained_workload_image("ctr-retained", "app:test") + .await + .expect("pin retained workload image"); + handle.await.unwrap(); + assert_eq!(resolved.id, "sha256:pinned-image-id"); + assert_eq!(resolved.oci_user, "1000:1000"); + let requests = requests.lock().unwrap(); + assert_eq!(requests.len(), 2, "unexpected requests: {requests:?}"); + assert!( + requests[0].contains("/libpod/containers/ctr-retained/json"), + "expected retained workload inspection first: {requests:?}" + ); + assert!( + requests[1].contains("/libpod/images/sha256%3Apinned-image-id/json"), + "expected pinned image inspection: {requests:?}" + ); + assert!( + !requests.iter().any(|request| request.contains("/pull")), + "supervisor-only refresh must never pull the workload tag: {requests:?}" + ); + } + + fn refresh_fixture( + sandbox_id: &str, + ) -> ( + ContainerListEntry, + DriverSandbox, + openshell_core::sandbox_generation::SandboxGenerationId, + ) { + use openshell_core::proto::compute::v1::DriverSandboxSpec; + let old_workload = ContainerListEntry { + id: format!("workload-{sandbox_id}"), + names: vec![format!("openshell-default--{sandbox_id}")], + state: "exited".to_string(), + labels: HashMap::from([(LABEL_SANDBOX_ID.to_string(), sandbox_id.to_string())]), + ports: None, + networks: None, + exit_code: 0, + }; + let durable = DriverSandbox { + id: sandbox_id.to_string(), + name: format!("sandbox-{sandbox_id}"), + namespace: String::new(), + spec: Some(DriverSandboxSpec::default()), + status: None, + workspace: String::new(), + }; + let generation = + openshell_core::sandbox_generation::SandboxGenerationId::parse("generation-1") + .expect("valid generation"); + (old_workload, durable, generation) + } + + const REFRESH_STOPPED_SUPERVISOR: &str = r#"{"Id":"sup-1","Name":"supervisor","State":{"Status":"exited","Running":false},"Config":{"Image":"sha256:supervisor-image"}}"#; + + #[tokio::test] + async fn supervisor_refresh_completes_fallible_prep_before_renaming() { + // A transient secret-lookup failure must return before the previous + // supervisor is renamed, so the canonical supervisor is never left + // under its rollback name where a normal delete cannot find it. + let (socket, requests, handle) = spawn_podman_stub( + "refresh-prep-before-rename", + vec![ + StubResponse::new(StatusCode::OK, REFRESH_STOPPED_SUPERVISOR), + StubResponse::new( + StatusCode::INTERNAL_SERVER_ERROR, + r#"{"message":"transient secret lookup failure"}"#, + ), + ], + ); + let driver = test_driver(socket); + let (old_workload, durable, generation) = refresh_fixture("prep-1"); + let result = driver + .replace_supervisor_for_network_trust(&old_workload, &durable, &generation, b"auth") + .await; + assert!( + result.is_err(), + "secret lookup failure must fail the refresh" + ); + handle.await.unwrap(); + let requests = requests.lock().unwrap(); + assert!( + !requests.iter().any(|request| request.contains("/rename")), + "fallible preparation must complete before any rename: {requests:?}" + ); + } + + #[tokio::test] + async fn supervisor_refresh_restores_canonical_name_when_replacement_fails() { + // The replacement create fails at its first image pull; the previous + // supervisor must be restored to the canonical name and must not be + // deleted, so a later retry observes the original stopped pair. + let (socket, requests, handle) = spawn_podman_stub( + "refresh-restore-on-failure", + vec![ + StubResponse::new(StatusCode::OK, REFRESH_STOPPED_SUPERVISOR), + StubResponse::new(StatusCode::NOT_FOUND, r#"{"message":"no secret"}"#), + StubResponse::new(StatusCode::NO_CONTENT, ""), + StubResponse::new( + StatusCode::INTERNAL_SERVER_ERROR, + r#"{"message":"image pull failed"}"#, + ), + StubResponse::new(StatusCode::NO_CONTENT, ""), + ], + ); + let driver = test_driver(socket); + let (old_workload, durable, generation) = refresh_fixture("restore-1"); + let result = driver + .replace_supervisor_for_network_trust(&old_workload, &durable, &generation, b"auth") + .await; + assert!(result.is_err(), "failed replacement must fail the refresh"); + handle.await.unwrap(); + let requests = requests.lock().unwrap(); + let renames = requests + .iter() + .filter(|request| request.contains("/rename")) + .collect::>(); + assert_eq!( + renames.len(), + 2, + "expected rename to rollback and rename back: {requests:?}" + ); + assert!( + renames[0].contains("openshell-supervisor-restore-1") + && renames[0].contains("openshell-supervisor-rollback-"), + "first rename moves the canonical supervisor to the rollback name: {renames:?}" + ); + assert!( + renames[1].contains("openshell-supervisor-rollback-") + && renames[1].contains("name=openshell-supervisor-restore-1"), + "second rename restores the canonical name: {renames:?}" + ); + assert!( + !requests + .iter() + .any(|request| request.starts_with("DELETE") && request.contains("rollback")), + "the restored supervisor must not be deleted: {requests:?}" + ); + } + + #[tokio::test] + async fn supervisor_refresh_names_orphaned_rollback_when_restore_fails() { + // If the restore rename itself fails, the error must name the + // rollback container so the orphan is identifiable for cleanup. + let (socket, _requests, handle) = spawn_podman_stub( + "refresh-restore-failure-orphan", + vec![ + StubResponse::new(StatusCode::OK, REFRESH_STOPPED_SUPERVISOR), + StubResponse::new(StatusCode::NOT_FOUND, r#"{"message":"no secret"}"#), + StubResponse::new(StatusCode::NO_CONTENT, ""), + StubResponse::new( + StatusCode::INTERNAL_SERVER_ERROR, + r#"{"message":"image pull failed"}"#, + ), + StubResponse::new( + StatusCode::INTERNAL_SERVER_ERROR, + r#"{"message":"rename raced with cleanup"}"#, + ), + ], + ); + let driver = test_driver(socket); + let (old_workload, durable, generation) = refresh_fixture("orphan-1"); + let error = driver + .replace_supervisor_for_network_trust(&old_workload, &durable, &generation, b"auth") + .await + .expect_err("restore failure must fail the refresh"); + handle.await.unwrap(); + let message = error.to_string(); + assert!( + message.contains("openshell-supervisor-rollback-"), + "error must name the orphaned rollback container: {message}" + ); + } + #[test] fn podman_driver_error_from_not_found() { let err = ComputeDriverError::from(PodmanApiError::NotFound("gone".into())); @@ -2302,7 +3013,10 @@ mod tests { let (start_socket, start_requests, start_handle) = spawn_podman_stub( "lifecycle-start", vec![ - StubResponse::new(StatusCode::OK, r#"[{"Id":"ctr-1","State":"stopped"}]"#), + StubResponse::new( + StatusCode::OK, + r#"[{"Id":"ctr-1","State":"stopped","Labels":{"openshell.ai/additional-ca-generation":"none"}}]"#, + ), StubResponse::new( StatusCode::OK, r#"{"Id":"ctr-1","Name":"sandbox","State":{"Status":"exited","Running":false,"FinishedAt":"2026-08-12T16:39:13Z"},"Config":{}}"#, @@ -2311,7 +3025,7 @@ mod tests { ); let authentication = encoded_launch_authentication(); test_driver(start_socket.clone()) - .start_sandbox("sandbox-1", "generation-1", &authentication) + .start_sandbox("sandbox-1", "generation-1", &authentication, None) .await .expect("start should succeed"); start_handle.await.expect("start stub should finish"); @@ -2499,6 +3213,7 @@ mod tests { "sandbox-1", "invalid-generation", b"secret-launch-authentication", + None, ) .with_subscriber(subscriber) .await @@ -2653,7 +3368,10 @@ mod tests { let (start_socket, _requests, start_handle) = spawn_podman_stub( "trace-start", vec![ - StubResponse::new(StatusCode::OK, r#"[{"Id":"ctr-1","State":"stopped"}]"#), + StubResponse::new( + StatusCode::OK, + r#"[{"Id":"ctr-1","State":"stopped","Labels":{"openshell.ai/additional-ca-generation":"none"}}]"#, + ), StubResponse::new( StatusCode::OK, r#"{"Id":"ctr-1","Name":"sandbox","State":{"Status":"exited","Running":false,"FinishedAt":"2026-08-12T16:39:13Z"},"Config":{}}"#, @@ -2662,7 +3380,7 @@ mod tests { ); let authentication = encoded_launch_authentication(); test_driver(start_socket.clone()) - .start_sandbox("sandbox-1", "generation-1", &authentication) + .start_sandbox("sandbox-1", "generation-1", &authentication, None) .with_subscriber(subscriber) .await .expect("start should succeed"); @@ -2921,6 +3639,7 @@ mod tests { } #[test] + fn local_podman_cdi_gpu_inventory_maps_nvidia_device_nodes() { let root = std::env::temp_dir().join(format!( "openshell-podman-gpu-test-{}-{}", @@ -4169,4 +4888,45 @@ mod tests { assert!(gpu.default_selection_supported); assert!(gpu.count_selection_supported); } + + #[test] + fn operator_network_trust_stages_immutable_artifact_from_readable_source() { + use std::os::unix::fs::PermissionsExt as _; + + let mut ca_params = rcgen::CertificateParams::new(Vec::::new()).unwrap(); + ca_params.is_ca = rcgen::IsCa::Ca(rcgen::BasicConstraints::Unconstrained); + let ca_key = openshell_crypto::pki::generate_keypair().unwrap(); + let ca = openshell_crypto::pki::self_signed(ca_params, &ca_key).unwrap(); + let source = std::env::temp_dir().join(format!( + "openshell-operator-ca-{}-{}.crt", + std::process::id(), + uuid::Uuid::new_v4() + )); + fs::write(&source, ca.pem()).unwrap(); + // The operator-owned source is deliberately writable `0644`, which + // the immutable-artifact verification must never see. + fs::set_permissions(&source, fs::Permissions::from_mode(0o644)).unwrap(); + + let (bundle, staging) = + operator_network_trust(source.to_str().unwrap()).expect("stage operator trust"); + let artifact = bundle.artifact_path(); + assert_ne!( + artifact, source, + "staged artifact must not be the source path" + ); + assert!(artifact.starts_with(staging.path())); + // The verification every create/start performs must accept the staged + // artifact without touching the operator's source permissions. + bundle + .verify_artifact() + .expect("staged operator artifact must satisfy immutable verification"); + let source_mode = fs::metadata(&source).unwrap().permissions().mode() & 0o777; + assert_eq!( + source_mode, 0o644, + "operator source permissions must be preserved" + ); + let artifact_mode = fs::metadata(artifact).unwrap().permissions().mode() & 0o777; + assert_eq!(artifact_mode, 0o444, "staged artifact must be read-only"); + let _ = fs::remove_file(&source); + } } diff --git a/crates/openshell-driver-podman/src/grpc.rs b/crates/openshell-driver-podman/src/grpc.rs index 6c73772a3..59b2fe1f6 100644 --- a/crates/openshell-driver-podman/src/grpc.rs +++ b/crates/openshell-driver-podman/src/grpc.rs @@ -187,23 +187,26 @@ impl ComputeDriver for ComputeDriverService { &self, request: Request, ) -> Result, Status> { - self.rpc_tracer - .trace(openshell_otel::rpc::START_SANDBOX, async { - let request = request.into_inner(); - if request.sandbox_id.is_empty() { - return Err(Status::invalid_argument("sandbox_id is required")); - } - self.driver - .start_sandbox( - &request.sandbox_id, - &request.generation_id, - &request.launch_authentication, - ) - .await - .map_err(Status::from)?; - Ok(Response::new(StartSandboxResponse::default())) - }) - .await + Box::pin( + self.rpc_tracer + .trace(openshell_otel::rpc::START_SANDBOX, async { + let request = request.into_inner(); + if request.sandbox_id.is_empty() { + return Err(Status::invalid_argument("sandbox_id is required")); + } + self.driver + .start_sandbox( + &request.sandbox_id, + &request.generation_id, + &request.launch_authentication, + request.sandbox.as_ref(), + ) + .await + .map_err(Status::from)?; + Ok(Response::new(StartSandboxResponse::default())) + }), + ) + .await } async fn delete_sandbox( diff --git a/crates/openshell-driver-podman/src/main.rs b/crates/openshell-driver-podman/src/main.rs index c0298197a..8a8057b2e 100644 --- a/crates/openshell-driver-podman/src/main.rs +++ b/crates/openshell-driver-podman/src/main.rs @@ -227,53 +227,58 @@ async fn main() -> Result<()> { }, ); - let driver = PodmanComputeDriver::new(PodmanComputeConfig { - allow_driver_config: args.admission_config_json.allow_driver_config, - resource_admission: args.admission_config_json.resource_admission.clone(), - socket_path: args.podman_socket, - default_image: args.sandbox_image.unwrap_or_default(), - image_pull_policy: args.sandbox_image_pull_policy, - grpc_endpoint: args.grpc_endpoint.unwrap_or_default(), - supervisor_otlp_endpoint: args.otlp_endpoint.clone(), - supervisor_gateway_name: args.gateway_name.clone(), - supervisor_ocsf_export: args.otlp_ocsf_endpoint.clone().map(|endpoint| { - openshell_core::sandbox_env::OcsfOtlpExport { - endpoint, - schema_version: args.otlp_ocsf_schema_version.clone(), - } - }), - gateway_port: args.gateway_port, - host_gateway_ip: args - .host_gateway_ip - .unwrap_or_else(PodmanComputeConfig::default_host_gateway_ip), - ssh_socket_path: args.sandbox_ssh_socket_path, - network_name: args.network_name, - stop_timeout_secs: args.stop_timeout, - sandbox_runtime_image: args - .sandbox_runtime_image - .unwrap_or_else(openshell_core::config::default_sandbox_runtime_image), - supervisor_image: args - .supervisor_image - .unwrap_or_else(openshell_core::config::default_supervisor_image), - guest_tls_ca: args.podman_tls_ca, - guest_tls_cert: args.podman_tls_cert, - guest_tls_key: args.podman_tls_key, - provider_spiffe_workload_api_socket: args.provider_spiffe_workload_api_socket, - app_armor_profile: args.app_armor_profile, - sandbox_pids_limit: args.sandbox_pids_limit, - health_check_interval_secs: args.health_check_interval_secs, - https_proxy: args.sandbox_https_proxy, - no_proxy: args.sandbox_no_proxy, - proxy_auth_file: args.sandbox_proxy_auth_file, - proxy_auth_allow_insecure: args.sandbox_proxy_auth_allow_insecure, - proxy_connect_by_hostname: args.sandbox_proxy_connect_by_hostname, - proxy_ca_bundle: args.sandbox_proxy_ca_bundle, - additional_ca_bundle: args.sandbox_additional_ca_bundle, - userns: args.userns, - uidmap: args.uidmap, - gidmap: args.gidmap, - enable_bind_mounts: args.enable_bind_mounts, - }) + // External drivers have no gateway-local staged artifact. Do not accept + // a path or silently emulate destination-CA delivery outside the gateway. + let driver = PodmanComputeDriver::new( + PodmanComputeConfig { + allow_driver_config: args.admission_config_json.allow_driver_config, + resource_admission: args.admission_config_json.resource_admission.clone(), + socket_path: args.podman_socket, + default_image: args.sandbox_image.unwrap_or_default(), + image_pull_policy: args.sandbox_image_pull_policy, + grpc_endpoint: args.grpc_endpoint.unwrap_or_default(), + supervisor_otlp_endpoint: args.otlp_endpoint.clone(), + supervisor_gateway_name: args.gateway_name.clone(), + supervisor_ocsf_export: args.otlp_ocsf_endpoint.clone().map(|endpoint| { + openshell_core::sandbox_env::OcsfOtlpExport { + endpoint, + schema_version: args.otlp_ocsf_schema_version.clone(), + } + }), + gateway_port: args.gateway_port, + host_gateway_ip: args + .host_gateway_ip + .unwrap_or_else(PodmanComputeConfig::default_host_gateway_ip), + ssh_socket_path: args.sandbox_ssh_socket_path, + network_name: args.network_name, + stop_timeout_secs: args.stop_timeout, + sandbox_runtime_image: args + .sandbox_runtime_image + .unwrap_or_else(openshell_core::config::default_sandbox_runtime_image), + supervisor_image: args + .supervisor_image + .unwrap_or_else(openshell_core::config::default_supervisor_image), + guest_tls_ca: args.podman_tls_ca, + guest_tls_cert: args.podman_tls_cert, + guest_tls_key: args.podman_tls_key, + provider_spiffe_workload_api_socket: args.provider_spiffe_workload_api_socket, + app_armor_profile: args.app_armor_profile, + sandbox_pids_limit: args.sandbox_pids_limit, + health_check_interval_secs: args.health_check_interval_secs, + https_proxy: args.sandbox_https_proxy, + no_proxy: args.sandbox_no_proxy, + proxy_auth_file: args.sandbox_proxy_auth_file, + proxy_auth_allow_insecure: args.sandbox_proxy_auth_allow_insecure, + proxy_connect_by_hostname: args.sandbox_proxy_connect_by_hostname, + proxy_ca_bundle: args.sandbox_proxy_ca_bundle, + additional_ca_bundle: args.sandbox_additional_ca_bundle, + userns: args.userns, + uidmap: args.uidmap, + gidmap: args.gidmap, + enable_bind_mounts: args.enable_bind_mounts, + }, + None, + ) .await .into_diagnostic()?; diff --git a/crates/openshell-driver-podman/src/watcher.rs b/crates/openshell-driver-podman/src/watcher.rs index 2163054a4..9a508e52c 100644 --- a/crates/openshell-driver-podman/src/watcher.rs +++ b/crates/openshell-driver-podman/src/watcher.rs @@ -681,17 +681,44 @@ mod tests { } fn podman_event(action: &str, sandbox_id: &str, time_nano: i64) -> PodmanEvent { + podman_event_for_container(action, sandbox_id, "container-1", time_nano) + } + + fn podman_event_for_container( + action: &str, + sandbox_id: &str, + container_id: &str, + time_nano: i64, + ) -> PodmanEvent { PodmanEvent { event_type: "container".to_string(), action: action.to_string(), actor: crate::client::EventActor { - id: "container-1".to_string(), + id: container_id.to_string(), attributes: HashMap::from([(LABEL_SANDBOX_ID.to_string(), sandbox_id.to_string())]), }, time_nano, } } + #[tokio::test] + async fn unrelated_remove_deletes_the_sandbox() { + let fences = LifecycleEventFences::default(); + let client = PodmanClient::new(crate::test_utils::unique_socket_path("watch-removals")); + + let event = map_podman_event( + &podman_event_for_container("remove", "sandbox-1", "external-container", 2), + &client, + &fences, + ) + .await + .expect("an unrelated remove must still delete the sandbox"); + assert!(matches!( + event.payload, + Some(watch_sandboxes_event::Payload::Deleted(_)) + )); + } + #[test] fn lifecycle_fence_rejects_delayed_stop_events_from_before_restart() { let fences = LifecycleEventFences::default(); diff --git a/crates/openshell-driver-vm/Cargo.toml b/crates/openshell-driver-vm/Cargo.toml index 691592c7c..6a757e40c 100644 --- a/crates/openshell-driver-vm/Cargo.toml +++ b/crates/openshell-driver-vm/Cargo.toml @@ -118,6 +118,7 @@ defaults-without-telemetry = ["compute-driver"] openshell-otel-test-support = { path = "../openshell-otel-test-support" } temp-env = "0.3" tempfile = "3" +rcgen = { workspace = true } opentelemetry_sdk = { workspace = true, features = ["testing"] } # smol-rs/polling drives the BSD/macOS parent-death detection in diff --git a/crates/openshell-driver-vm/src/driver.rs b/crates/openshell-driver-vm/src/driver.rs index d4905c3a4..29930d98b 100644 --- a/crates/openshell-driver-vm/src/driver.rs +++ b/crates/openshell-driver-vm/src/driver.rs @@ -27,6 +27,7 @@ use crate::rootfs::{ validate_host_supervisor, write_rootfs_image_file, }; use crate::runtime::VmBackend; +use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD}; use bollard::Docker; use bollard::errors::Error as BollardError; use bollard::models::ContainerCreateBody; @@ -71,6 +72,9 @@ use openshell_core::proto::compute::v1::{ use openshell_core::proto_struct::{ deserialize_optional_non_empty_string_list, struct_to_json_value, }; +use openshell_core::{ + NetworkSupervisorTrustBundle, network_trust::MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES, +}; use openshell_sandbox_backend::boundary_protocol::{ BoundaryConfig, BoundaryListener, GatewayVerificationKey, SandboxRuntimeDescriptor, SandboxTlsClientConfig, SandboxTlsMaterial, SandboxTlsServerConfig, SandboxTransport, @@ -86,11 +90,9 @@ use std::fs; use std::future::Future; use std::io::{BufRead, BufReader, BufWriter, Read, Seek, SeekFrom, Write}; #[cfg(unix)] -use std::os::fd::AsRawFd as _; +use std::os::fd::{AsRawFd as _, OwnedFd}; #[cfg(unix)] -use std::os::fd::OwnedFd; -#[cfg(unix)] -use std::os::unix::fs::PermissionsExt; +use std::os::unix::fs::{OpenOptionsExt as _, PermissionsExt}; #[cfg(unix)] use std::os::unix::process::ExitStatusExt as _; use std::path::{Component, Path, PathBuf}; @@ -213,6 +215,15 @@ const GUEST_BOUNDARY_CONFIG_ENV: &str = "OPENSHELL_VM_SANDBOX_BOOTSTRAP"; const HOST_AUTH_BUNDLE_FILE: &str = "supervisor-auth.json"; const HOST_RUNTIME_DESCRIPTOR_FILE: &str = "runtime-descriptor.json"; const HOST_BOUNDARY_GENERATION_FILE: &str = "boundary-generation"; +/// Per-sandbox host-only copy of the gateway-normalized destination trust. +/// +/// This file is deliberately outside the VM disks. The host supervisor is the +/// only process which receives its path; the guest runs `openshell-sandbox` +/// and must never receive this material through its overlay, rootfs, argv, or +/// environment. +const HOST_ADDITIONAL_CA_FILE: &str = "additional-ca.crt"; +const HOST_ADDITIONAL_CA_TEMP_PREFIX: &str = ".additional-ca."; +const HOST_ADDITIONAL_CA_TEMP_SUFFIX: &str = ".tmp"; /// The backend this driver admits. VM-specific placement remains inside the /// opaque runtime descriptor. const DRIVER_ADMITTED_BACKEND: &str = openshell_sandbox_backend::BACKEND_NAME; @@ -257,6 +268,7 @@ const IMAGE_PREP_CONSOLE_LOG: &str = "image-prep-console.log"; const PREPARED_IMAGE_ROOTFS_DIR: &str = "/image-rootfs"; static IMAGE_CACHE_BUILD_COUNTER: AtomicU64 = AtomicU64::new(0); static OWNER_STATE_WRITE_COUNTER: AtomicU64 = AtomicU64::new(0); +static NETWORK_TRUST_WRITE_COUNTER: AtomicU64 = AtomicU64::new(0); #[derive(Debug, Clone)] struct VmDriverTlsPaths { @@ -290,6 +302,41 @@ enum GuestImagePayloadSource { LocalDocker { rootfs_archive: PathBuf }, } +/// Driver-owned immutable copy of gateway-normalized destination trust. +/// +/// This is public only because [`VmDriverConfig`] is public. Its contents are +/// intentionally not constructible outside this module, and +/// [`VmDriver::new`] rejects a value supplied by a caller. The VM driver alone +/// creates it after reading and verifying the gateway artifact once. +#[derive(Clone)] +pub struct VmNetworkSupervisorTrustSnapshot { + bundle: NetworkSupervisorTrustBundle, +} + +impl VmNetworkSupervisorTrustSnapshot { + fn normalized_pem(&self) -> &[u8] { + self.bundle.normalized_pem() + } + + fn digest(&self) -> &str { + self.bundle.digest() + } + + fn certificate_count(&self) -> usize { + self.bundle.certificate_count() + } +} + +/// Redact PEM bytes and their original gateway artifact path from diagnostics. +impl std::fmt::Debug for VmNetworkSupervisorTrustSnapshot { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("VmNetworkSupervisorTrustSnapshot") + .field("certificate_count", &self.certificate_count()) + .field("digest", &self.digest()) + .finish_non_exhaustive() + } +} + #[derive(Clone, serde::Serialize, serde::Deserialize)] #[serde(deny_unknown_fields)] #[allow(clippy::struct_excessive_bools)] @@ -323,7 +370,22 @@ pub struct VmDriverConfig { pub guest_tls_ca: Option, pub guest_tls_cert: Option, pub guest_tls_key: Option, - /// Corporate forward proxy settings delivered to the guest init script. + /// Gateway-owned, normalized destination-trust artifact path received on + /// the private gateway-to-driver argv. It is consumed exactly once during + /// driver initialization and then cleared. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub additional_ca_bundle: Option, + /// SHA-256 generation expected for `additional_ca_bundle`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub additional_ca_digest: Option, + /// Number of canonical certificate blocks expected in the artifact. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub additional_ca_certificate_count: Option, + /// Internal immutable trust state. Callers must leave this unset; it is + /// deliberately excluded from all serialized configuration input/output. + #[serde(skip)] + pub additional_ca_snapshot: Option, + /// Corporate forward proxy settings delivered to the host supervisor. #[serde(flatten)] pub upstream_proxy: UpstreamProxyConfig, /// Gateway-host PEM CA bundle staged into the guest overlay for the @@ -382,6 +444,10 @@ impl std::fmt::Debug for VmDriverConfig { .field("guest_tls_ca", &self.guest_tls_ca) .field("guest_tls_cert", &self.guest_tls_cert) .field("guest_tls_key", &self.guest_tls_key) + .field( + "additional_ca_configured", + &(self.additional_ca_bundle.is_some() || self.additional_ca_snapshot.is_some()), + ) .field("gpu_enabled", &self.gpu_enabled) .field("gpu_mem_mib", &self.gpu_mem_mib) .field("gpu_vcpus", &self.gpu_vcpus) @@ -452,6 +518,10 @@ impl Default for VmDriverConfig { guest_tls_ca: None, guest_tls_cert: None, guest_tls_key: None, + additional_ca_bundle: None, + additional_ca_digest: None, + additional_ca_certificate_count: None, + additional_ca_snapshot: None, upstream_proxy: UpstreamProxyConfig::default(), proxy_ca_bundle: None, provider_spiffe_workload_api_tcp_endpoint: None, @@ -598,6 +668,353 @@ impl VmDriverConfig { } } +/// Consume the gateway's private launch metadata and retain only a +/// driver-owned immutable trust snapshot. +/// +/// The standalone VM driver cannot receive the bundle bytes over the gateway +/// RPC, so the in-process gateway passes a path, digest, and count on its +/// authenticated child-process argv. Treat those three values as one atomic +/// contract: accepting a partial set would make it possible to read a mutable +/// path without an authenticated generation or to silently lose a trust +/// anchor. The source is read only here; launches use the resulting bytes and +/// never re-open the gateway artifact. +fn initialize_additional_ca_snapshot(config: &mut VmDriverConfig) -> Result<(), String> { + if config.additional_ca_snapshot.is_some() { + return Err( + "additional CA snapshot is driver-owned and must not be supplied by callers" + .to_string(), + ); + } + + let snapshot = match ( + config.additional_ca_bundle.as_ref(), + config.additional_ca_digest.as_deref(), + config.additional_ca_certificate_count, + ) { + (None, None, None) => None, + (Some(path), Some(expected_digest), Some(expected_certificate_count)) => { + if path.as_os_str().is_empty() { + return Err("additional CA artifact path must not be empty".to_string()); + } + if expected_certificate_count == 0 { + return Err("additional CA certificate count must be greater than zero".to_string()); + } + let expected_digest = normalized_additional_ca_digest(expected_digest)?; + let pem = read_gateway_additional_ca_artifact(path)?; + // Deterministic destination-trust generation label, not an + // authenticity proof: the equality check below only detects + // staged-artifact drift; trust comes from the gateway's + // normalization and TLS verification. Classified as a non-security + // identity hash in openshell-crypto/README.md. + let actual_digest = format!("sha256:{:x}", Sha256::digest(&pem)); + if actual_digest != expected_digest { + return Err(format!( + "additional CA artifact '{}' does not match expected digest {}", + path.display(), + expected_digest + )); + } + let actual_certificate_count = normalized_additional_ca_certificate_count(&pem) + .map_err(|error| { + format!( + "additional CA artifact '{}' is not a normalized certificate bundle: {error}", + path.display() + ) + })?; + if actual_certificate_count != expected_certificate_count { + return Err(format!( + "additional CA artifact '{}' has certificate count {}, expected {}", + path.display(), + actual_certificate_count, + expected_certificate_count + )); + } + Some(VmNetworkSupervisorTrustSnapshot { + // The bundle's artifact path records provenance only. Never + // call `verify_artifact` on this retained value: all later + // launches must use these captured immutable bytes rather + // than rereading the mutable gateway-host source path. + bundle: NetworkSupervisorTrustBundle::new( + pem, + actual_certificate_count, + expected_digest, + path.clone(), + ), + }) + } + _ => { + return Err( + "additional CA artifact path, digest, and certificate count must be supplied together" + .to_string(), + ); + } + }; + + // Do not retain the mutable gateway artifact path or metadata after the + // authenticated, bounded startup read. In particular this makes a later + // sandbox launch incapable of accidentally rereading it. + config.additional_ca_bundle = None; + config.additional_ca_digest = None; + config.additional_ca_certificate_count = None; + config.additional_ca_snapshot = snapshot; + Ok(()) +} + +/// Require the generation form emitted by gateway normalization. +fn normalized_additional_ca_digest(digest: &str) -> Result { + let Some(hex) = digest.strip_prefix("sha256:") else { + return Err("additional CA digest must use the sha256: prefix".to_string()); + }; + if hex.len() != 64 || !hex.bytes().all(|byte| byte.is_ascii_hexdigit()) { + return Err("additional CA digest must be a SHA-256 digest".to_string()); + } + if hex.bytes().any(|byte| byte.is_ascii_uppercase()) { + return Err("additional CA digest must use normalized lowercase hexadecimal".to_string()); + } + Ok(format!("sha256:{hex}")) +} + +/// Read a normalized gateway artifact once without following symlinks. +fn read_gateway_additional_ca_artifact(path: &Path) -> Result, String> { + let metadata = fs::symlink_metadata(path).map_err(|error| { + format!( + "inspect additional CA artifact '{}': {error}", + path.display() + ) + })?; + if !metadata.file_type().is_file() { + return Err(format!( + "additional CA artifact '{}' is not a regular file", + path.display() + )); + } + validate_gateway_additional_ca_artifact_metadata(path, &metadata)?; + + let mut options = fs::OpenOptions::new(); + options.read(true); + #[cfg(unix)] + { + // The pre-open metadata check gives portable diagnostics, while these + // flags ensure an attacker cannot replace it with a symlink, FIFO, or + // blocking special file before the actual one-time read. + options.custom_flags(libc::O_CLOEXEC | libc::O_NOFOLLOW | libc::O_NONBLOCK); + } + let mut file = options + .open(path) + .map_err(|error| format!("open additional CA artifact '{}': {error}", path.display()))?; + let opened_metadata = file.metadata().map_err(|error| { + format!( + "inspect additional CA artifact '{}': {error}", + path.display() + ) + })?; + if !opened_metadata.file_type().is_file() { + return Err(format!( + "additional CA artifact '{}' is not a regular file", + path.display() + )); + } + validate_gateway_additional_ca_artifact_metadata(path, &opened_metadata)?; + + let capacity = usize::try_from(opened_metadata.len()).map_err(|_| { + format!( + "additional CA artifact '{}' exceeds the {}-byte limit", + path.display(), + MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES + ) + })?; + let mut pem = Vec::with_capacity(capacity); + Read::by_ref(&mut file) + .take((MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES + 1) as u64) + .read_to_end(&mut pem) + .map_err(|error| format!("read additional CA artifact '{}': {error}", path.display()))?; + if pem.len() > MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES { + return Err(format!( + "additional CA artifact '{}' exceeds the {}-byte limit", + path.display(), + MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES + )); + } + Ok(pem) +} + +fn validate_gateway_additional_ca_artifact_metadata( + path: &Path, + metadata: &fs::Metadata, +) -> Result<(), String> { + if metadata.len() > MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES as u64 { + return Err(format!( + "additional CA artifact '{}' exceeds the {}-byte limit", + path.display(), + MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES + )); + } + #[cfg(unix)] + if metadata.permissions().mode() & 0o7777 != 0o444 { + return Err(format!( + "additional CA artifact '{}' does not have the required read-only permissions", + path.display() + )); + } + Ok(()) +} + +/// Verify that the bytes are exactly the canonical PEM representation emitted +/// by the gateway's destination-trust normalization boundary, and return its +/// number of certificate blocks. +fn normalized_additional_ca_certificate_count(pem: &[u8]) -> Result { + let text = std::str::from_utf8(pem).map_err(|_| "bundle is not UTF-8 PEM text")?; + let mut remainder = text; + let mut normalized = Vec::with_capacity(pem.len()); + let mut certificate_count = 0_usize; + + while !remainder.is_empty() { + let Some(after_begin) = remainder.strip_prefix("-----BEGIN CERTIFICATE-----\n") else { + return Err("bundle contains material outside canonical certificate blocks"); + }; + remainder = after_begin; + let mut encoded = String::new(); + loop { + let Some(newline) = remainder.find('\n') else { + return Err("certificate block is missing its terminating newline"); + }; + let line = &remainder[..newline]; + remainder = &remainder[newline + 1..]; + if line == "-----END CERTIFICATE-----" { + break; + } + if line.is_empty() || line.len() > 64 { + return Err("certificate block has a non-canonical base64 line"); + } + encoded.push_str(line); + } + let decoded = BASE64_STANDARD + .decode(encoded.as_bytes()) + .map_err(|_| "certificate block contains invalid base64")?; + if decoded.is_empty() { + return Err("certificate block is empty"); + } + normalized.extend_from_slice(b"-----BEGIN CERTIFICATE-----\n"); + let canonical = BASE64_STANDARD.encode(decoded); + for line in canonical.as_bytes().chunks(64) { + normalized.extend_from_slice(line); + normalized.push(b'\n'); + } + normalized.extend_from_slice(b"-----END CERTIFICATE-----\n"); + certificate_count += 1; + } + + if certificate_count == 0 { + return Err("bundle contains no certificate blocks"); + } + if normalized != pem { + return Err("bundle is not canonical normalized PEM"); + } + Ok(certificate_count) +} + +/// Write or remove the host-only per-sandbox trust artifact. +/// +/// The snapshot has already authenticated the bytes against the gateway's +/// generation. Do not replace this with a source-path copy: re-opening that +/// path here would let a changed gateway artifact alter later sandbox +/// launches. The temporary file is created with restrictive permissions and +/// atomically published into the sandbox's private state directory. +fn stage_additional_ca_snapshot( + state_dir: &Path, + snapshot: Option<&VmNetworkSupervisorTrustSnapshot>, +) -> Result, String> { + let destination = state_dir.join(HOST_ADDITIONAL_CA_FILE); + let Some(snapshot) = snapshot else { + remove_additional_ca_material(state_dir, &destination)?; + return Ok(None); + }; + + let temporary = state_dir.join(format!( + "{HOST_ADDITIONAL_CA_TEMP_PREFIX}{}-{}{}", + std::process::id(), + NETWORK_TRUST_WRITE_COUNTER.fetch_add(1, Ordering::Relaxed), + HOST_ADDITIONAL_CA_TEMP_SUFFIX, + )); + let result: Result<(), String> = (|| { + let mut options = fs::OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + options.mode(0o600); + let mut file = options + .open(&temporary) + .map_err(|error| format!("create host additional CA temporary file: {error}"))?; + file.write_all(snapshot.normalized_pem()) + .map_err(|error| format!("write host additional CA temporary file: {error}"))?; + file.sync_all() + .map_err(|error| format!("sync host additional CA temporary file: {error}"))?; + #[cfg(unix)] + fs::set_permissions(&temporary, fs::Permissions::from_mode(0o600)) + .map_err(|error| format!("restrict host additional CA temporary file: {error}"))?; + fs::rename(&temporary, &destination) + .map_err(|error| format!("publish host additional CA file: {error}"))?; + fs::File::open(state_dir) + .and_then(|directory| directory.sync_all()) + .map_err(|error| format!("sync host additional CA directory: {error}"))?; + Ok(()) + })(); + if result.is_err() { + let _ = fs::remove_file(&temporary); + } + result?; + Ok(Some(destination)) +} + +/// Remove both the published host artifact and interrupted-write remnants. +fn remove_additional_ca_material(state_dir: &Path, destination: &Path) -> Result<(), String> { + remove_additional_ca_file(destination)?; + let entries = fs::read_dir(state_dir) + .map_err(|error| format!("read sandbox state while clearing additional CA: {error}"))?; + for entry in entries { + let entry = entry.map_err(|error| { + format!("read sandbox state entry while clearing additional CA: {error}") + })?; + let name = entry.file_name(); + let name = name.to_string_lossy(); + if name.starts_with(HOST_ADDITIONAL_CA_TEMP_PREFIX) + && name.ends_with(HOST_ADDITIONAL_CA_TEMP_SUFFIX) + { + remove_additional_ca_file(&entry.path())?; + } + } + Ok(()) +} + +fn remove_additional_ca_file(path: &Path) -> Result<(), String> { + match fs::remove_file(path) { + Ok(()) => Ok(()), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(error) => Err(format!( + "remove host additional CA file '{}': {error}", + path.display() + )), + } +} + +/// Render the host-supervisor-only arguments for a staged trust snapshot. +/// +/// PEM bytes never appear in argv. The supervisor receives only the +/// restrictive host file path and gateway-authenticated generation digest. +fn additional_ca_supervisor_args( + snapshot: Option<&VmNetworkSupervisorTrustSnapshot>, + staged_path: Option<&Path>, +) -> Result, String> { + match (snapshot, staged_path) { + (None, None) => Ok(Vec::new()), + (Some(_snapshot), Some(path)) => Ok(vec![ + "--additional-ca-bundle".to_string(), + path.display().to_string(), + ]), + (Some(_), None) => Err("additional CA snapshot was not staged".to_string()), + (None, Some(_)) => Err("staged additional CA exists without a snapshot".to_string()), + } +} + fn validate_openshell_endpoint(endpoint: &str) -> Result<(), String> { let url = Url::parse(endpoint) .map_err(|err| format!("invalid openshell endpoint '{endpoint}': {err}"))?; @@ -766,6 +1183,7 @@ impl VmDriver { } validate_openshell_endpoint(&config.grpc_endpoint)?; let _ = config.tls_paths()?; + initialize_additional_ca_snapshot(&mut config)?; config.state_dir = absolute_state_dir(&config.state_dir)?; #[cfg(target_os = "linux")] @@ -935,6 +1353,24 @@ impl VmDriver { Ok(destination) } + /// Stage the captured destination trust into this sandbox's host-only + /// state directory. This intentionally happens per launch so stopped + /// sandboxes do not carry a stale source generation into a later host + /// supervisor process. + async fn stage_additional_ca_for_sandbox( + &self, + state_dir: &Path, + ) -> Result, Status> { + let state_dir = state_dir.to_path_buf(); + let snapshot = self.config.additional_ca_snapshot.clone(); + tokio::task::spawn_blocking(move || { + stage_additional_ca_snapshot(&state_dir, snapshot.as_ref()) + }) + .await + .map_err(|error| Status::internal(format!("stage additional CA task panicked: {error}")))? + .map_err(|error| Status::internal(format!("stage host additional CA: {error}"))) + } + async fn spawn_host_supervisor( &self, sandbox: &Sandbox, @@ -944,6 +1380,15 @@ impl VmDriver { auth_bundle: &openshell_core::jwt::SupervisorAuthBundle, sandbox_owner: SandboxOwnerIdentity, ) -> Result<(Child, Option), Status> { + // The destination CA is staged on the host before either resolving + // the supervisor binary or spawning it. It never crosses into a VM + // disk or a guest environment. + let staged_network_trust = self.stage_additional_ca_for_sandbox(state_dir).await?; + let network_trust_args = additional_ca_supervisor_args( + self.config.additional_ca_snapshot.as_ref(), + staged_network_trust.as_deref(), + ) + .map_err(Status::internal)?; let supervisor_binary = self.host_supervisor_binary().await?; let (openshell_endpoint, gateway_tls_server_name) = host_control_openshell_endpoint(&self.config.grpc_endpoint) @@ -1004,6 +1449,7 @@ impl VmDriver { .arg("--workdir") .arg("/sandbox") .args(upstream_proxy_args) + .args(network_trust_args) .env( openshell_core::sandbox_env::ADMITTED_ISOLATION_BACKEND, DRIVER_ADMITTED_BACKEND, @@ -7872,8 +8318,9 @@ mod tests { GpuResourceRequirements, ResourceRequirements, }; use prost_types::{Struct, Value, value::Kind}; + use rcgen::{BasicConstraints, CertificateParams, IsCa, KeyPair}; use std::fs; - use std::path::Path; + use std::path::{Path, PathBuf}; use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering}; use std::time::{SystemTime, UNIX_EPOCH}; use tonic::Code; @@ -7994,6 +8441,283 @@ mod tests { assert!(error.to_string().contains("openshell_endpoint")); } + fn normalized_test_network_ca_bundle(certificate_count: usize) -> Vec { + let mut pem = String::new(); + for _ in 0..certificate_count { + let mut params = CertificateParams::new(Vec::::new()).unwrap(); + params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained); + let key = KeyPair::generate().unwrap(); + pem.push_str(¶ms.self_signed(&key).unwrap().pem()); + } + pem.into_bytes() + } + + fn write_gateway_network_ca_artifact(path: &Path, pem: &[u8]) { + let _ = fs::remove_file(path); + fs::write(path, pem).unwrap(); + #[cfg(unix)] + fs::set_permissions(path, fs::Permissions::from_mode(0o444)).unwrap(); + } + + fn network_ca_config(path: PathBuf, pem: &[u8], certificate_count: usize) -> VmDriverConfig { + VmDriverConfig { + grpc_endpoint: "http://127.0.0.1:8080".to_string(), + additional_ca_bundle: Some(path), + additional_ca_digest: Some(format!("sha256:{:x}", Sha256::digest(pem))), + additional_ca_certificate_count: Some(certificate_count), + ..Default::default() + } + } + + #[test] + fn additional_ca_launch_options_are_all_or_none_and_snapshot_is_driver_owned() { + let path = PathBuf::from("/gateway/artifact/additional-ca.crt"); + let digest = format!("sha256:{}", "a".repeat(64)); + for config in [ + VmDriverConfig { + additional_ca_bundle: Some(path.clone()), + ..Default::default() + }, + VmDriverConfig { + additional_ca_digest: Some(digest.clone()), + ..Default::default() + }, + VmDriverConfig { + additional_ca_certificate_count: Some(1), + ..Default::default() + }, + VmDriverConfig { + additional_ca_bundle: Some(path.clone()), + additional_ca_digest: Some(digest.clone()), + ..Default::default() + }, + VmDriverConfig { + additional_ca_bundle: Some(path), + additional_ca_certificate_count: Some(1), + ..Default::default() + }, + VmDriverConfig { + additional_ca_digest: Some(digest), + additional_ca_certificate_count: Some(1), + ..Default::default() + }, + ] { + let mut config = config; + let error = initialize_additional_ca_snapshot(&mut config) + .expect_err("partial private launch metadata must be rejected"); + assert!(error.contains("must be supplied together"), "{error}"); + } + + let mut no_trust = VmDriverConfig::default(); + initialize_additional_ca_snapshot(&mut no_trust) + .expect("all absent is the explicit no-trust state"); + assert!(no_trust.additional_ca_snapshot.is_none()); + + let supplied_snapshot = VmNetworkSupervisorTrustSnapshot { + bundle: NetworkSupervisorTrustBundle::new( + b"-----BEGIN CERTIFICATE-----\nprivate-fixture-bytes\n-----END CERTIFICATE-----\n" + .to_vec(), + 1, + format!("sha256:{}", "b".repeat(64)), + PathBuf::from("/untrusted/caller-snapshot"), + ), + }; + let mut caller_snapshot = VmDriverConfig { + additional_ca_snapshot: Some(supplied_snapshot.clone()), + ..Default::default() + }; + let error = initialize_additional_ca_snapshot(&mut caller_snapshot) + .expect_err("callers must not preload driver-owned snapshot bytes"); + assert!(error.contains("driver-owned"), "{error}"); + + let result = VmDriver::new(VmDriverConfig { + grpc_endpoint: "http://127.0.0.1:8080".to_string(), + bootstrap_image: "bootstrap:latest".to_string(), + additional_ca_snapshot: Some(supplied_snapshot), + ..Default::default() + }); + let error = futures::executor::block_on(result) + .err() + .expect("driver initialization must reject caller-provided snapshot state"); + assert!(error.contains("driver-owned"), "{error}"); + } + + #[test] + fn additional_ca_snapshot_verifies_normalized_digest_and_certificate_count() { + let directory = tempfile::tempdir().unwrap(); + let artifact = directory.path().join("gateway-additional-ca.crt"); + let pem = normalized_test_network_ca_bundle(1); + write_gateway_network_ca_artifact(&artifact, &pem); + + let mut accepted = network_ca_config(artifact.clone(), &pem, 1); + initialize_additional_ca_snapshot(&mut accepted) + .expect("gateway-normalized artifact should produce a snapshot"); + let snapshot = accepted.additional_ca_snapshot.as_ref().expect("snapshot"); + assert_eq!(snapshot.normalized_pem(), pem); + assert_eq!(snapshot.certificate_count(), 1); + assert!(accepted.additional_ca_bundle.is_none()); + assert!(accepted.additional_ca_digest.is_none()); + assert!(accepted.additional_ca_certificate_count.is_none()); + + let mut non_normalized_digest = network_ca_config(artifact.clone(), &pem, 1); + non_normalized_digest.additional_ca_digest = Some(format!("sha256:{}", "A".repeat(64))); + let error = initialize_additional_ca_snapshot(&mut non_normalized_digest) + .expect_err("digest must use normalized lowercase hexadecimal"); + assert!(error.contains("normalized lowercase"), "{error}"); + + let tampered = b"private replacement CA bytes must not be disclosed\n"; + fs::remove_file(&artifact).unwrap(); + fs::write(&artifact, tampered).unwrap(); + #[cfg(unix)] + fs::set_permissions(&artifact, fs::Permissions::from_mode(0o444)).unwrap(); + let mut tampered_config = network_ca_config(artifact.clone(), &pem, 1); + let error = initialize_additional_ca_snapshot(&mut tampered_config) + .expect_err("a changed gateway artifact must fail closed"); + assert!(error.contains("does not match expected digest"), "{error}"); + assert!(!error.contains("private replacement"), "{error}"); + + write_gateway_network_ca_artifact(&artifact, &pem); + let mut wrong_count = network_ca_config(artifact, &pem, 2); + let error = initialize_additional_ca_snapshot(&mut wrong_count) + .expect_err("wrong certificate count must fail closed"); + assert!(error.contains("certificate count 1, expected 2"), "{error}"); + } + + #[cfg(unix)] + #[test] + fn additional_ca_snapshot_read_is_bounded_and_never_follows_symlinks() { + use std::os::unix::fs::symlink; + + let directory = tempfile::tempdir().unwrap(); + let too_large = directory.path().join("too-large.crt"); + let bytes = vec![b'x'; MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES + 1]; + write_gateway_network_ca_artifact(&too_large, &bytes); + let mut oversized = network_ca_config(too_large, &bytes, 1); + let error = initialize_additional_ca_snapshot(&mut oversized) + .expect_err("oversized gateway artifact must be rejected before reading it"); + assert!(error.contains("exceeds the"), "{error}"); + + let target = directory.path().join("target.crt"); + let target_pem = normalized_test_network_ca_bundle(1); + write_gateway_network_ca_artifact(&target, &target_pem); + let link = directory.path().join("symlink.crt"); + symlink(&target, &link).unwrap(); + let mut symlinked = network_ca_config(link, &target_pem, 1); + let error = initialize_additional_ca_snapshot(&mut symlinked) + .expect_err("gateway artifact symlink must not be followed"); + assert!(error.contains("not a regular file"), "{error}"); + } + + #[test] + fn additional_ca_staging_uses_immutable_snapshot_and_host_supervisor_argv() { + let directory = tempfile::tempdir().unwrap(); + let artifact = directory.path().join("gateway-additional-ca.crt"); + let pem = normalized_test_network_ca_bundle(1); + write_gateway_network_ca_artifact(&artifact, &pem); + let mut config = network_ca_config(artifact.clone(), &pem, 1); + initialize_additional_ca_snapshot(&mut config).unwrap(); + let snapshot = config.additional_ca_snapshot.as_ref().unwrap(); + + // Change the original path after initialization. Staging must copy the + // retained Arc-backed bytes rather than reread this mutable source. + fs::remove_file(&artifact).unwrap(); + fs::write( + &artifact, + b"private source replacement must not reach a launch\n", + ) + .unwrap(); + #[cfg(unix)] + fs::set_permissions(&artifact, fs::Permissions::from_mode(0o444)).unwrap(); + + let state_dir = directory.path().join("sandbox-state"); + fs::create_dir(&state_dir).unwrap(); + let staged = stage_additional_ca_snapshot(&state_dir, Some(snapshot)) + .expect("stage captured bytes") + .expect("configured snapshot stages a host file"); + assert_eq!(fs::read(&staged).unwrap(), pem); + #[cfg(unix)] + assert_eq!( + fs::metadata(&staged).unwrap().permissions().mode() & 0o777, + 0o600 + ); + assert!( + !state_dir.join("overlay.ext4").exists(), + "destination trust must never be written to a guest overlay" + ); + + let args = additional_ca_supervisor_args(Some(snapshot), Some(&staged)).unwrap(); + assert_eq!( + args, + ["--additional-ca-bundle", staged.to_string_lossy().as_ref(),] + ); + let diagnostic = format!("{config:?} {snapshot:?} {args:?}"); + assert!( + !diagnostic.contains("private source replacement"), + "{diagnostic}" + ); + assert!( + !diagnostic.contains("BEGIN CERTIFICATE"), + "PEM bytes must not appear in diagnostics or argv: {diagnostic}" + ); + + let guest_env = build_guest_environment( + &Sandbox { + id: "network-ca-sandbox".to_string(), + ..Default::default() + }, + &config, + ); + assert!( + !guest_env + .iter() + .any(|entry| entry.contains("additional-ca") + || entry.contains("BEGIN CERTIFICATE")), + "destination trust must not enter guest launch environment: {guest_env:?}" + ); + } + + #[test] + fn additional_ca_staging_clears_stale_host_material_for_no_trust() { + let directory = tempfile::tempdir().unwrap(); + let artifact = directory.path().join("gateway-additional-ca.crt"); + let pem = normalized_test_network_ca_bundle(1); + write_gateway_network_ca_artifact(&artifact, &pem); + let mut config = network_ca_config(artifact, &pem, 1); + initialize_additional_ca_snapshot(&mut config).unwrap(); + let snapshot = config.additional_ca_snapshot.as_ref().unwrap(); + let state_dir = directory.path().join("sandbox-state"); + fs::create_dir(&state_dir).unwrap(); + + let staged = stage_additional_ca_snapshot(&state_dir, Some(snapshot)) + .unwrap() + .unwrap(); + assert!(staged.is_file()); + let stale_temporary = state_dir.join(format!( + "{HOST_ADDITIONAL_CA_TEMP_PREFIX}stale{HOST_ADDITIONAL_CA_TEMP_SUFFIX}" + )); + fs::write(&stale_temporary, b"stale trust bytes").unwrap(); + + assert_eq!( + stage_additional_ca_snapshot(&state_dir, None).unwrap(), + None + ); + assert!(!staged.exists()); + assert!(!stale_temporary.exists()); + assert!( + additional_ca_supervisor_args(None, None) + .unwrap() + .is_empty() + ); + + // A subsequent configured generation recreates only the host file; + // no guest-overlay state participates in either transition. + let recreated = stage_additional_ca_snapshot(&state_dir, Some(snapshot)) + .unwrap() + .unwrap(); + assert_eq!(fs::read(recreated).unwrap(), pem); + assert!(!state_dir.join("overlay.ext4").exists()); + } + struct TestTracing { exporter: opentelemetry_sdk::trace::InMemorySpanExporter, _provider: opentelemetry_sdk::trace::SdkTracerProvider, diff --git a/crates/openshell-driver-vm/src/main.rs b/crates/openshell-driver-vm/src/main.rs index fb7c4accf..63fa3540b 100644 --- a/crates/openshell-driver-vm/src/main.rs +++ b/crates/openshell-driver-vm/src/main.rs @@ -142,6 +142,22 @@ struct Args { #[arg(long = "guest-tls-key", env = "OPENSHELL_VM_TLS_KEY")] guest_tls_key: Option, + /// Gateway-owned normalized destination trust artifact. This hidden option + /// is supplied only by the in-process gateway VM factory. + #[arg(long, hide = true)] + additional_ca_bundle: Option, + + /// Expected SHA-256 generation of the gateway-owned destination trust + /// artifact. Supplied with the path so the child can authenticate its + /// one-time startup read without receiving CA bytes on argv or in env. + #[arg(long, hide = true)] + additional_ca_digest: Option, + + /// Expected certificate count in the gateway-normalized destination trust + /// artifact. + #[arg(long, hide = true)] + additional_ca_certificate_count: Option, + /// Corporate forward proxy for supervisor TLS egress. #[arg(long, env = "OPENSHELL_VM_UPSTREAM_PROXY")] upstream_proxy: Option, @@ -338,6 +354,10 @@ async fn main() -> Result<()> { guest_tls_ca: args.guest_tls_ca.clone(), guest_tls_cert: args.guest_tls_cert.clone(), guest_tls_key: args.guest_tls_key.clone(), + additional_ca_bundle: args.additional_ca_bundle.clone(), + additional_ca_digest: args.additional_ca_digest.clone(), + additional_ca_certificate_count: args.additional_ca_certificate_count, + additional_ca_snapshot: None, upstream_proxy: openshell_core::UpstreamProxyConfig { https_proxy: args.upstream_proxy.clone(), no_proxy: args.upstream_no_proxy.clone(), @@ -827,6 +847,45 @@ mod tests { assert!(args.upstream_proxy_ca_bundle.is_none()); } + #[test] + fn internal_network_trust_options_have_no_environment_aliases() { + let args = Args::parse_from([ + "openshell-driver-vm", + "--additional-ca-bundle", + "/var/lib/openshell/network/additional-ca.crt", + "--additional-ca-digest", + "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + "--additional-ca-certificate-count", + "2", + ]); + assert_eq!( + args.additional_ca_bundle.as_deref(), + Some(PathBuf::from("/var/lib/openshell/network/additional-ca.crt").as_path()) + ); + assert_eq!( + args.additional_ca_digest.as_deref(), + Some("sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef") + ); + assert_eq!(args.additional_ca_certificate_count, Some(2)); + + temp_env::with_vars( + [ + ( + "OPENSHELL_VM_ADDITIONAL_CA_BUNDLE", + Some("/tmp/untrusted.crt"), + ), + ("OPENSHELL_VM_ADDITIONAL_CA_DIGEST", Some("sha256:bad")), + ("OPENSHELL_VM_ADDITIONAL_CA_CERTIFICATE_COUNT", Some("999")), + ], + || { + let args = Args::parse_from(["openshell-driver-vm"]); + assert!(args.additional_ca_bundle.is_none()); + assert!(args.additional_ca_digest.is_none()); + assert!(args.additional_ca_certificate_count.is_none()); + }, + ); + } + #[test] fn peer_authorization_accepts_matching_uid_and_pid() { authorize_peer_credentials( diff --git a/crates/openshell-gateway/Cargo.toml b/crates/openshell-gateway/Cargo.toml index fa977a2c2..86249568b 100644 --- a/crates/openshell-gateway/Cargo.toml +++ b/crates/openshell-gateway/Cargo.toml @@ -79,5 +79,7 @@ vendored-z3 = ["openshell-server/vendored-z3"] workspace = true [dev-dependencies] +openshell-crypto = { workspace = true, default-features = false, features = ["aws-lc"] } +rcgen = { workspace = true } tempfile = "3" toml = { workspace = true } diff --git a/crates/openshell-gateway/src/lib.rs b/crates/openshell-gateway/src/lib.rs index 3f060ae9f..d86e1d257 100644 --- a/crates/openshell-gateway/src/lib.rs +++ b/crates/openshell-gateway/src/lib.rs @@ -265,6 +265,10 @@ impl openshell_server::ComputeDriverFactory for KubernetesFactory { true } + fn supports_network_supervisor_trust(&self) -> bool { + true + } + fn validate_config( &self, context: openshell_server::ComputeDriverConfigContext<'_>, @@ -287,8 +291,19 @@ impl openshell_server::ComputeDriverFactory for KubernetesFactory { config.supervisor_gateway_name = Some(context.gateway_name().to_string()); // With the gateway supervisor topology, the gateway owns the shared // supervisor's assignments and token; the driver adapts them to - // Kubernetes objects. + // Kubernetes objects. The experimental shared supervisor topology + // does not support the network supervisor additional destination CA; + // refuse the combination fail-closed rather than silently + // supervising generations without the operator's trust bundle. + let network_trust = context.network_trust_bundle().cloned(); let driver = if config.shared_supervisor.is_some() { + if network_trust.is_some() { + return Err(openshell_core::Error::execution( + "the experimental shared supervisor topology does not support the \ + network supervisor additional destination CA" + .to_string(), + )); + } openshell_driver_kubernetes::KubernetesComputeDriver::with_shared_supervision( config, context.shutdown_receiver(), @@ -299,6 +314,7 @@ impl openshell_server::ComputeDriverFactory for KubernetesFactory { openshell_driver_kubernetes::KubernetesComputeDriver::new( config, context.shutdown_receiver(), + network_trust, ) .await } @@ -344,6 +360,10 @@ impl openshell_server::ComputeDriverFactory for DockerFactory { true } + fn supports_network_supervisor_trust(&self) -> bool { + true + } + fn validate_config( &self, context: openshell_server::ComputeDriverConfigContext<'_>, @@ -369,6 +389,7 @@ impl openshell_server::ComputeDriverFactory for DockerFactory { context.gateway_bind_address(), context.gateway_log_level(), &config, + context.network_trust_bundle().cloned(), ) .await .map_err(|error| openshell_core::Error::execution(error.to_string()))?; @@ -410,6 +431,10 @@ impl openshell_server::ComputeDriverFactory for PodmanFactory { true } + fn supports_network_supervisor_trust(&self) -> bool { + true + } + fn validate_config( &self, context: openshell_server::ComputeDriverConfigContext<'_>, @@ -432,9 +457,12 @@ impl openshell_server::ComputeDriverFactory for PodmanFactory { config.supervisor_gateway_name = Some(context.gateway_name().to_string()); require_guest_tls_for_local_driver(&context, "podman")?; apply_guest_tls(&mut config.guest_tls_ca, context.guest_tls_ca()); - let driver = openshell_driver_podman::PodmanComputeDriver::new(config) - .await - .map_err(|error| openshell_core::Error::execution(error.to_string()))?; + let driver = openshell_driver_podman::PodmanComputeDriver::new( + config, + context.network_trust_bundle().cloned(), + ) + .await + .map_err(|error| openshell_core::Error::execution(error.to_string()))?; let driver = openshell_driver_podman::ComputeDriverService::new_in_process(driver); Ok(openshell_server::ComputeDriverInstance::InProcess( std::sync::Arc::new(driver), @@ -482,6 +510,10 @@ impl openshell_server::ComputeDriverFactory for VmFactory { .map_err(openshell_core::Error::config) } + fn supports_network_supervisor_trust(&self) -> bool { + true + } + fn validate_config( &self, context: openshell_server::ComputeDriverConfigContext<'_>, @@ -520,6 +552,7 @@ impl openshell_server::ComputeDriverFactory for VmFactory { context.gateway_name(), &config, context.otlp_config(), + context.network_trust_bundle(), ) .await?; Ok(openshell_server::ComputeDriverInstance::ManagedRemote( @@ -704,4 +737,17 @@ mod tests { expected ); } + + #[cfg(not(target_os = "windows"))] + #[test] + fn first_party_network_supervisor_drivers_opt_into_trust_propagation() { + #[cfg(feature = "compute-driver-docker")] + assert!(::supports_network_supervisor_trust(&DockerFactory)); + #[cfg(feature = "compute-driver-kubernetes")] + assert!(::supports_network_supervisor_trust(&KubernetesFactory)); + #[cfg(feature = "compute-driver-podman")] + assert!(::supports_network_supervisor_trust(&PodmanFactory)); + #[cfg(feature = "compute-driver-vm")] + assert!(::supports_network_supervisor_trust(&VmFactory)); + } } diff --git a/crates/openshell-gateway/src/vm.rs b/crates/openshell-gateway/src/vm.rs index 149b143ee..95a4a85b9 100644 --- a/crates/openshell-gateway/src/vm.rs +++ b/crates/openshell-gateway/src/vm.rs @@ -31,7 +31,7 @@ use hyper_util::rt::TokioIo; use openshell_core::proto::compute::v1::{ GetCapabilitiesRequest, compute_driver_client::ComputeDriverClient, }; -use openshell_core::{Error, Result, UpstreamProxyConfig}; +use openshell_core::{Error, NetworkSupervisorTrustBundle, Result, UpstreamProxyConfig}; #[cfg(unix)] use openshell_otel::TraceContextInterceptor; use openshell_server::AcquiredRemoteDriverEndpoint; @@ -531,6 +531,7 @@ pub async fn spawn( gateway_name: &str, vm_config: &VmComputeConfig, otlp_config: Option<&OtlpConfig>, + additional_ca_bundle: Option<&NetworkSupervisorTrustBundle>, ) -> Result { vm_config.validate_configuration()?; let driver_bin = resolve_compute_driver_bin(vm_config)?; @@ -580,6 +581,7 @@ pub async fn spawn( command.arg("--guest-tls-ca").arg(tls.ca); } append_vm_proxy_and_spiffe_args(&mut command, vm_config); + append_vm_network_trust_args(&mut command, additional_ca_bundle); if vm_config.enable_bind_mounts { command.arg("--enable-bind-mounts"); } @@ -638,6 +640,25 @@ fn append_vm_rootfs_tar_args(command: &mut Command, config: &VmComputeConfig) { } } +#[cfg(unix)] +fn append_vm_network_trust_args( + command: &mut Command, + bundle: Option<&NetworkSupervisorTrustBundle>, +) { + if let Some(bundle) = bundle { + // Only stable, non-secret artifact metadata crosses the process + // boundary. The child reads and verifies the bounded artifact once, + // then retains its own immutable snapshot for sandbox launches. + command + .arg("--additional-ca-bundle") + .arg(bundle.artifact_path()); + command.arg("--additional-ca-digest").arg(bundle.digest()); + command + .arg("--additional-ca-certificate-count") + .arg(bundle.certificate_count().to_string()); + } +} + #[cfg(unix)] fn append_vm_proxy_and_spiffe_args(command: &mut Command, config: &VmComputeConfig) { let proxy = &config.upstream_proxy; @@ -694,6 +715,7 @@ pub async fn spawn( _gateway_name: &str, _vm_config: &VmComputeConfig, _otlp_config: Option<&OtlpConfig>, + _additional_ca_bundle: Option<&NetworkSupervisorTrustBundle>, ) -> Result { Err(Error::config( "the vm compute driver requires unix domain socket support", @@ -775,13 +797,13 @@ async fn connect_compute_driver(socket_path: &Path) -> Result { #[cfg(all(test, unix))] mod tests { use super::{ - VmComputeConfig, append_otlp_args, append_vm_identity_args, + VmComputeConfig, append_otlp_args, append_vm_identity_args, append_vm_network_trust_args, append_vm_proxy_and_spiffe_args, append_vm_rootfs_tar_args, compute_driver_guest_tls_paths, compute_driver_socket_path, current_euid, prepare_compute_driver_socket_path, prepare_vm_state_dir, resolve_compute_driver_bin, resolve_driver_search_dirs, validate_vm_sandbox_identity, }; - use openshell_core::UpstreamProxyConfig; + use openshell_core::{NetworkSupervisorTrustBundle, UpstreamProxyConfig}; use openshell_server::config_file::{ OcsfOtlpConfig, OcsfSchemaVersion, OtlpConfig, OtlpTracesConfig, }; @@ -943,6 +965,45 @@ mod tests { assert_eq!(command.as_std().get_args().count(), 0); } + #[test] + fn vm_driver_command_forwards_only_network_trust_artifact_metadata() { + let mut command = tokio::process::Command::new("openshell-driver-vm"); + let pem = + b"-----BEGIN CERTIFICATE-----\nprivate-fixture-bytes\n-----END CERTIFICATE-----\n"; + let bundle = NetworkSupervisorTrustBundle::new( + pem.to_vec(), + 2, + "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + PathBuf::from("/var/lib/openshell/network/additional-ca.crt"), + ); + append_vm_network_trust_args(&mut command, Some(&bundle)); + let args = command + .as_std() + .get_args() + .map(|arg| arg.to_string_lossy().into_owned()) + .collect::>(); + assert_eq!( + args, + [ + "--additional-ca-bundle", + "/var/lib/openshell/network/additional-ca.crt", + "--additional-ca-digest", + "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + "--additional-ca-certificate-count", + "2", + ] + ); + assert!(!args.iter().any(|arg| arg == "OPENSHELL_TLS_CA")); + assert!( + !args.iter().any(|arg| arg.contains("private-fixture-bytes")), + "CA material must not cross the process boundary in argv: {args:?}" + ); + + let mut unset = tokio::process::Command::new("openshell-driver-vm"); + append_vm_network_trust_args(&mut unset, None); + assert_eq!(unset.as_std().get_args().count(), 0); + } + #[test] fn invalid_corporate_proxy_config_is_rejected_before_the_driver_starts() { let err = UpstreamProxyConfig { diff --git a/crates/openshell-gateway/tests/network_trust_startup.rs b/crates/openshell-gateway/tests/network_trust_startup.rs new file mode 100644 index 000000000..c4d5674f0 --- /dev/null +++ b/crates/openshell-gateway/tests/network_trust_startup.rs @@ -0,0 +1,238 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Process-level startup coverage for global network-supervisor trust material. + +#![cfg(unix)] + +use std::fs; +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; + +use openshell_crypto::pki::generate_simple_self_signed; +use tempfile::TempDir; + +const CONFIG_FIELD: &str = "openshell.supervisor.network.additional_ca_bundle"; + +struct GatewayFixture { + root: TempDir, + config_path: PathBuf, +} + +impl GatewayFixture { + fn new(source_path: &Path) -> Self { + let root = tempfile::tempdir().expect("create gateway fixture directory"); + let config_path = root.path().join("gateway.toml"); + fs::write( + &config_path, + format!( + "[openshell]\nversion = 2\n\n[openshell.supervisor.network]\nadditional_ca_bundle = \"{}\"\n", + source_path.display() + ), + ) + .expect("write gateway config"); + Self { root, config_path } + } + + fn run(&self, driver: &str, socket: Option<&Path>) -> Output { + let mut command = Command::new(env!("CARGO_BIN_EXE_openshell-gateway")); + command + .args([ + "--config", + self.config_path.to_str().expect("UTF-8 config path"), + "--db-url", + "sqlite::memory:", + "--disable-tls", + "--compute-driver", + driver, + "--log-level", + "info", + ]) + .env("XDG_STATE_HOME", self.root.path().join("state")) + .env("XDG_CONFIG_HOME", self.root.path().join("config")) + .env_remove("RUST_LOG") + .env_remove("OPENSHELL_DRIVERS") + .env_remove("OPENSHELL_COMPUTE_DRIVER") + .env_remove("OPENSHELL_COMPUTE_DRIVER_SOCKET") + .env_remove("OPENSHELL_GATEWAY_CONFIG"); + if let Some(socket) = socket { + command.args([ + "--compute-driver-socket", + socket.to_str().expect("UTF-8 socket path"), + ]); + } + command.output().expect("run openshell-gateway") + } +} + +fn combined_output(output: &Output) -> String { + let raw = format!( + "{}{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + let raw = raw.as_bytes(); + let mut plain = Vec::with_capacity(raw.len()); + let mut index = 0; + while index < raw.len() { + if raw[index] == 0x1b { + index += 1; + while index < raw.len() { + let byte = raw[index]; + index += 1; + if byte.is_ascii_alphabetic() { + break; + } + } + } else { + plain.push(raw[index]); + index += 1; + } + } + String::from_utf8_lossy(&plain) + .replace('│', " ") + .split_whitespace() + .collect::>() + .join(" ") +} + +fn pem_payload_marker(pem: &str) -> &str { + pem.lines() + .find(|line| line.len() > 32 && !line.starts_with("-----")) + .expect("PEM fixture payload line") +} + +/// Substring check that ignores the gateway CLI's diagnostic line wrapping. +/// +/// The rendered diagnostic wraps at a fixed width with `│` continuation +/// markers, and long tokens (for example source paths) are hard-broken +/// mid-word. Squashing whitespace and continuation glyphs on both sides +/// keeps assertions independent of where the wrap lands. +fn contains_wrapped(haystack: &str, needle: &str) -> bool { + fn squash(text: &str) -> String { + text.chars() + .filter(|c| !c.is_whitespace() && *c != '│') + .collect() + } + squash(haystack).contains(&squash(needle)) +} + +fn generated_certificate() -> openshell_crypto::pki::CertifiedKey { + generate_simple_self_signed(vec!["destination.example".to_string()]) + .expect("generate certificate fixture") +} + +#[test] +fn valid_bundle_reports_only_redacted_metadata_before_custom_driver_rejection() { + let source_dir = tempfile::tempdir().expect("create source directory"); + let certificate = generated_certificate(); + let source_path = source_dir.path().join("destination-ca.pem"); + let pem = certificate.cert.pem(); + fs::write(&source_path, &pem).expect("write certificate fixture"); + let fixture = GatewayFixture::new(&source_path); + + let output = fixture.run("custom-driver", None); + let diagnostic = combined_output(&output); + + assert!(!output.status.success(), "custom driver must be rejected"); + assert!( + contains_wrapped( + &diagnostic, + "Network supervisor additional destination trust configured" + ), + "missing startup metadata in:\n{diagnostic}" + ); + assert!( + contains_wrapped(&diagnostic, "certificate_count=1"), + "{diagnostic}" + ); + assert!(contains_wrapped(&diagnostic, "sha256:"), "{diagnostic}"); + assert!(contains_wrapped(&diagnostic, CONFIG_FIELD), "{diagnostic}"); + assert!( + contains_wrapped(&diagnostic, "custom-driver"), + "{diagnostic}" + ); + assert!(contains_wrapped(&diagnostic, "unsupported"), "{diagnostic}"); + assert!( + !contains_wrapped(&diagnostic, pem_payload_marker(&pem)), + "certificate payload leaked" + ); +} + +#[test] +fn invalid_bundles_fail_startup_with_actionable_redacted_errors() { + let source_dir = tempfile::tempdir().expect("create source directory"); + let generated = generated_certificate(); + let private_key = generated.key_pair.serialize_pem().unwrap(); + let cases = [ + ( + source_dir.path().join("missing.pem"), + None, + "could not be read", + ), + ( + source_dir.path().join("empty.pem"), + Some(String::new()), + "contains no PEM certificate", + ), + ( + source_dir.path().join("private-key.pem"), + Some(private_key.clone()), + "contains a non-certificate PEM", + ), + ]; + + for (source_path, contents, expected) in cases { + if let Some(contents) = contents { + fs::write(&source_path, contents).expect("write invalid fixture"); + } + let fixture = GatewayFixture::new(&source_path); + let output = fixture.run("docker", None); + let diagnostic = combined_output(&output); + + assert!(!output.status.success(), "invalid bundle must fail startup"); + assert!(contains_wrapped(&diagnostic, CONFIG_FIELD), "{diagnostic}"); + assert!( + contains_wrapped(&diagnostic, &source_path.display().to_string()), + "{diagnostic}" + ); + assert!(contains_wrapped(&diagnostic, expected), "{diagnostic}"); + assert!( + !contains_wrapped(&diagnostic, pem_payload_marker(&private_key)), + "private key payload leaked" + ); + assert!( + !contains_wrapped(&diagnostic, "Starting OpenShell server"), + "driver/server startup began after invalid material:\n{diagnostic}" + ); + } +} + +#[test] +fn configured_remote_endpoint_is_rejected_before_connection() { + let source_dir = tempfile::tempdir().expect("create source directory"); + let certificate = generated_certificate(); + let source_path = source_dir.path().join("destination-ca.pem"); + fs::write(&source_path, certificate.cert.pem()).expect("write certificate fixture"); + let fixture = GatewayFixture::new(&source_path); + let socket_path = fixture.root.path().join("must-not-connect.sock"); + + let output = fixture.run("docker", Some(&socket_path)); + let diagnostic = combined_output(&output); + + assert!(!output.status.success(), "remote override must be rejected"); + assert!(contains_wrapped(&diagnostic, CONFIG_FIELD), "{diagnostic}"); + assert!( + contains_wrapped(&diagnostic, "unsupported remote"), + "{diagnostic}" + ); + assert!(contains_wrapped(&diagnostic, "endpoint"), "{diagnostic}"); + assert!( + !contains_wrapped(&diagnostic, "failed to create compute runtime"), + "gateway attempted remote-driver construction:\n{diagnostic}" + ); + assert!( + !contains_wrapped(&diagnostic, "No such file or directory"), + "gateway attempted to connect to the sentinel socket:\n{diagnostic}" + ); +} diff --git a/crates/openshell-sandbox/src/child_env.rs b/crates/openshell-sandbox/src/child_env.rs index 50549a743..44d6504b6 100644 --- a/crates/openshell-sandbox/src/child_env.rs +++ b/crates/openshell-sandbox/src/child_env.rs @@ -3,6 +3,42 @@ use std::path::Path; +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum TlsEnvironmentMode { + /// `OpenShell`'s generated trust files are authoritative for mediated traffic. + Override, + /// Preserve caller-provided TLS variables and fill only missing values. + FillMissing, +} + +const TLS_ENVIRONMENT_VARIABLES: [&str; 6] = [ + "NODE_EXTRA_CA_CERTS", + "DENO_CERT", + "SSL_CERT_FILE", + "REQUESTS_CA_BUNDLE", + "CURL_CA_BUNDLE", + "GIT_SSL_CAINFO", +]; + +#[must_use] +pub fn is_tls_environment_variable(key: &str) -> bool { + TLS_ENVIRONMENT_VARIABLES.contains(&key) +} + +pub fn tls_env_vars_for_mode( + ca_cert_path: &Path, + combined_bundle_path: &Path, + mode: TlsEnvironmentMode, + user_environment: &std::collections::HashMap, +) -> Vec<(&'static str, String)> { + tls_env_vars(ca_cert_path, combined_bundle_path) + .into_iter() + .filter(|(key, _)| { + matches!(mode, TlsEnvironmentMode::Override) || !user_environment.contains_key(*key) + }) + .collect() +} + pub fn tls_env_vars( ca_cert_path: &Path, combined_bundle_path: &Path, @@ -49,5 +85,15 @@ mod tests { assert!(stdout.contains("REQUESTS_CA_BUNDLE=/etc/openshell-tls/ca-bundle.pem")); assert!(stdout.contains("CURL_CA_BUNDLE=/etc/openshell-tls/ca-bundle.pem")); assert!(stdout.contains("GIT_SSL_CAINFO=/etc/openshell-tls/ca-bundle.pem")); + + let keys = tls_env_vars(ca_cert_path, combined_bundle_path).map(|(key, _)| key); + assert!( + !keys.contains(&openshell_core::sandbox_env::TLS_CA), + "destination/child trust must not overwrite gateway mTLS trust" + ); + assert_ne!( + ca_cert_path, + Path::new(openshell_core::container_paths::TLS_CA_MOUNT_PATH) + ); } } diff --git a/crates/openshell-sandbox/src/delegated.rs b/crates/openshell-sandbox/src/delegated.rs index 144d3679a..bf712d912 100644 --- a/crates/openshell-sandbox/src/delegated.rs +++ b/crates/openshell-sandbox/src/delegated.rs @@ -86,6 +86,9 @@ pub async fn spawn_workload( launcher.clone(), )); + // The delegated workload runs behind OpenShell's mediated network path, + // so the generated trust files remain authoritative for its TLS settings. + let tls_environment_mode = crate::child_env::TlsEnvironmentMode::Override; #[cfg(target_os = "linux")] let mut handle = ProcessHandle::spawn( launcher, @@ -95,6 +98,7 @@ pub async fn spawn_workload( interactive, policy, ca_file_paths.as_ref(), + tls_environment_mode, &provider_env, ) .wrap_err("spawn delegated workload process")?; @@ -106,6 +110,7 @@ pub async fn spawn_workload( interactive, policy, ca_file_paths.as_ref(), + tls_environment_mode, &provider_env, )?; diff --git a/crates/openshell-sandbox/src/process.rs b/crates/openshell-sandbox/src/process.rs index 1b0f853d2..9aeae4884 100644 --- a/crates/openshell-sandbox/src/process.rs +++ b/crates/openshell-sandbox/src/process.rs @@ -428,6 +428,7 @@ impl ProcessHandle { interactive: bool, policy: &SandboxPolicy, ca_paths: Option<&(PathBuf, PathBuf)>, + tls_environment_mode: child_env::TlsEnvironmentMode, provider_env: &HashMap, ) -> Result { Self::spawn_impl( @@ -438,6 +439,7 @@ impl ProcessHandle { interactive, policy, ca_paths, + tls_environment_mode, provider_env, ) } @@ -456,6 +458,7 @@ impl ProcessHandle { interactive: bool, policy: &SandboxPolicy, ca_paths: Option<&(PathBuf, PathBuf)>, + tls_environment_mode: child_env::TlsEnvironmentMode, provider_env: &HashMap, ) -> Result { Self::spawn_impl( @@ -465,6 +468,7 @@ impl ProcessHandle { interactive, policy, ca_paths, + tls_environment_mode, provider_env, ) } @@ -479,6 +483,7 @@ impl ProcessHandle { interactive: bool, policy: &SandboxPolicy, ca_paths: Option<&(PathBuf, PathBuf)>, + tls_environment_mode: child_env::TlsEnvironmentMode, provider_env: &HashMap, ) -> Result { let mut cmd = Command::new(program); @@ -511,12 +516,13 @@ impl ProcessHandle { // inherited environment. The entrypoint drops to the sandbox user // before `exec`; without this strip, sandbox code could recover // supervisor credentials from its inherited environment. + let user_environment = configured_user_environment(); apply_canonical_process_environment( &mut cmd, policy, workspace, interactive, - &configured_user_environment(), + &user_environment, ); strip_supervisor_only_env(&mut cmd); @@ -528,9 +534,28 @@ impl ProcessHandle { strip_proxy_env(&mut cmd); - // Set TLS trust store env vars so sandbox processes trust the ephemeral CA + // In interception mode OpenShell trust is authoritative. In direct + // additional-CA mode, retain all caller values and only fill gaps. if let Some((ca_cert_path, combined_bundle_path)) = ca_paths { - for (key, value) in child_env::tls_env_vars(ca_cert_path, combined_bundle_path) { + // Provider credentials are injected before this point. Reassert + // caller TLS values in additive mode so a provider environment + // cannot accidentally replace a caller-selected trust store. + if matches!( + tls_environment_mode, + child_env::TlsEnvironmentMode::FillMissing + ) { + for (key, value) in &user_environment { + if child_env::is_tls_environment_variable(key) { + cmd.env(key, value); + } + } + } + for (key, value) in child_env::tls_env_vars_for_mode( + ca_cert_path, + combined_bundle_path, + tls_environment_mode, + &user_environment, + ) { cmd.env(key, value); } } @@ -639,6 +664,7 @@ impl ProcessHandle { interactive: bool, policy: &SandboxPolicy, ca_paths: Option<&(PathBuf, PathBuf)>, + tls_environment_mode: child_env::TlsEnvironmentMode, provider_env: &HashMap, ) -> Result { let mut cmd = Command::new(program); @@ -671,12 +697,13 @@ impl ProcessHandle { // Strip supervisor-only identity material from the entrypoint's // inherited environment. + let user_environment = configured_user_environment(); apply_canonical_process_environment( &mut cmd, policy, workspace, interactive, - &configured_user_environment(), + &user_environment, ); strip_supervisor_only_env(&mut cmd); @@ -688,9 +715,28 @@ impl ProcessHandle { strip_proxy_env(&mut cmd); - // Set TLS trust store env vars so sandbox processes trust the ephemeral CA + // In interception mode OpenShell trust is authoritative. In direct + // additional-CA mode, retain all caller values and only fill gaps. if let Some((ca_cert_path, combined_bundle_path)) = ca_paths { - for (key, value) in child_env::tls_env_vars(ca_cert_path, combined_bundle_path) { + // Provider credentials are injected before this point. Reassert + // caller TLS values in additive mode so a provider environment + // cannot accidentally replace a caller-selected trust store. + if matches!( + tls_environment_mode, + child_env::TlsEnvironmentMode::FillMissing + ) { + for (key, value) in &user_environment { + if child_env::is_tls_environment_variable(key) { + cmd.env(key, value); + } + } + } + for (key, value) in child_env::tls_env_vars_for_mode( + ca_cert_path, + combined_bundle_path, + tls_environment_mode, + &user_environment, + ) { cmd.env(key, value); } } diff --git a/crates/openshell-server/src/cli.rs b/crates/openshell-server/src/cli.rs index 8c15bd896..7c5d5b111 100644 --- a/crates/openshell-server/src/cli.rs +++ b/crates/openshell-server/src/cli.rs @@ -401,6 +401,15 @@ fn prepare_server_config_with_drivers( } else { None }; + // Normalize sources now, but defer artifact staging until the selected + // driver has been proven able to propagate the bundle. This keeps a + // rejected startup/preflight from leaving gateway state behind. + let normalized_network_trust = file + .as_ref() + .map(|file| crate::network_trust::normalize_from_config(&file.openshell.supervisor.network)) + .transpose() + .map_err(|error| miette::miette!("{error}"))? + .flatten(); if let Some(file) = file.as_ref() { merge_file_into_args(args, &file.openshell.gateway, matches); } @@ -665,10 +674,46 @@ fn prepare_server_config_with_drivers( // user-authored config file. config.gateway_jwt = gateway_jwt; + if let Some(normalized) = normalized_network_trust.as_ref() { + // Use the same resolver as runtime construction, with the explicit + // configuration marker but no synthetic/staged artifact. This rejects + // remote endpoints and factories that have not opted into trust + // propagation before any state is written. + let driver_startup = crate::compute::driver_config::DriverStartupContext { + file: file.as_ref(), + guest_tls: guest_tls.as_ref(), + network_trust: None, + network_trust_configured: true, + gateway_port: config.bind_address.port(), + gateway_tls_enabled: config.tls.is_some(), + endpoint_overrides: &config.compute_driver_endpoints, + }; + crate::validate_compute_driver_config( + compute_drivers, + compute_driver.name(), + &config.name, + config.bind_address, + &config.log_level, + driver_startup, + false, + ) + .map_err(|error| { + miette::miette!( + "Network supervisor additional destination trust configured: certificate_count={} digest={}: {error}", + normalized.certificate_count(), + normalized.digest() + ) + })?; + } + let network_trust = normalized_network_trust + .map(crate::network_trust::stage_normalized) + .transpose()?; + Ok(ServerStartupConfig { config, config_file: file, guest_tls, + network_trust, compute_driver, legacy_compute_driver_env_seen, }) @@ -748,6 +793,13 @@ async fn run_from_args( if prepared.legacy_compute_driver_env_seen { warn!("OPENSHELL_DRIVERS is deprecated; migrate to OPENSHELL_COMPUTE_DRIVER"); } + if let Some(bundle) = prepared.network_trust.as_ref() { + info!( + certificate_count = bundle.certificate_count(), + digest = bundle.digest(), + "Network supervisor additional destination trust configured" + ); + } let has_client_ca = prepared .config @@ -833,6 +885,10 @@ impl crate::ComputeDriverFactory for PreflightTestFactory { true } + fn supports_network_supervisor_trust(&self) -> bool { + true + } + async fn build( &self, _context: crate::ComputeDriverBuildContext<'_>, @@ -932,6 +988,12 @@ fn run_effective_config_preflight( let empty_file = ConfigFile::default(); let semantic_file = file.as_ref().unwrap_or(&empty_file); validate_preflight_semantics(&run, matches, semantic_file)?; + // Preflight must exercise the same full source parsing as startup, + // while deliberately leaving state artifacts untouched. + let network_trust_configured = crate::network_trust::normalize_from_config( + &semantic_file.openshell.supervisor.network, + )? + .is_some(); let mut endpoint_overrides = BTreeMap::new(); if let Some(selection) = selection.as_ref() @@ -942,6 +1004,8 @@ fn run_effective_config_preflight( let driver_startup = crate::compute::driver_config::DriverStartupContext { file: file.as_ref(), guest_tls: None, + network_trust: None, + network_trust_configured, gateway_port: run.port, gateway_tls_enabled: !run.disable_tls, endpoint_overrides: &endpoint_overrides, @@ -1539,6 +1603,43 @@ mod tests { } } + static ENV_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); + + struct EnvVarGuard { + key: &'static str, + original: Option, + } + + impl EnvVarGuard { + #[allow(unsafe_code)] + fn set(key: &'static str, value: &str) -> Self { + let original = std::env::var(key).ok(); + // SAFETY: tests serialize environment mutation with ENV_LOCK. + unsafe { std::env::set_var(key, value) }; + Self { key, original } + } + + #[allow(unsafe_code)] + fn remove(key: &'static str) -> Self { + let original = std::env::var(key).ok(); + // SAFETY: tests serialize environment mutation with ENV_LOCK. + unsafe { std::env::remove_var(key) }; + Self { key, original } + } + } + + impl Drop for EnvVarGuard { + #[allow(unsafe_code)] + fn drop(&mut self) { + match self.original.as_deref() { + // SAFETY: tests serialize environment mutation with ENV_LOCK. + Some(value) => unsafe { std::env::set_var(self.key, value) }, + // SAFETY: tests serialize environment mutation with ENV_LOCK. + None => unsafe { std::env::remove_var(self.key) }, + } + } + } + static REGISTRY_DETECTION_CALLS: AtomicUsize = AtomicUsize::new(0); static REJECTING_VALIDATION_CALLS: AtomicUsize = AtomicUsize::new(0); @@ -1563,6 +1664,10 @@ mod tests { true } + fn supports_network_supervisor_trust(&self) -> bool { + true + } + async fn build( &self, _context: crate::ComputeDriverBuildContext<'_>, @@ -2546,6 +2651,53 @@ mod tests { }); } + #[test] + fn config_preflight_normalizes_network_trust_without_creating_an_artifact() { + let _lock = ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let config_home = tempfile::tempdir().unwrap(); + let source_home = tempfile::tempdir().unwrap(); + let state_parent = tempfile::tempdir().unwrap(); + let state_home = state_parent.path().join("not-created"); + let source = source_home.path().join("destination-ca.pem"); + let certificate = rcgen::generate_simple_self_signed(vec!["destination.example".into()]) + .expect("test certificate"); + std::fs::write(&source, certificate.cert.pem()).unwrap(); + let config = config_home.path().join("gateway.toml"); + std::fs::write( + &config, + format!( + "[openshell]\nversion = 2\n\n[openshell.supervisor.network]\nadditional_ca_bundle = \"{}\"\n", + source.display() + ), + ) + .unwrap(); + let _config_env = EnvVarGuard::remove("OPENSHELL_GATEWAY_CONFIG"); + let _state = EnvVarGuard::set("XDG_STATE_HOME", state_home.to_str().unwrap()); + let (run, matches) = parse_with_args(&[ + "openshell-gateway", + "--compute-driver", + "test", + "--disable-tls", + ]); + + super::run_config_preflight( + super::ConfigPreflightArgs { + path: Some(config), + ..Default::default() + }, + run, + &matches, + ) + .expect("valid trust source and capable selected driver pass preflight"); + + assert!( + !state_home.exists(), + "preflight must not create a normalized trust artifact" + ); + } + #[test] fn config_preflight_explicit_path_overrides_environment_selection() { let dir = tempfile::tempdir().unwrap(); @@ -3017,6 +3169,107 @@ mod tests { }); } + #[test] + fn prepare_server_config_normalizes_global_network_trust_before_driver_startup() { + let _lock = ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let state = tempfile::tempdir().unwrap(); + let source_dir = tempfile::tempdir().unwrap(); + let config_dir = tempfile::tempdir().unwrap(); + let _state = EnvVarGuard::set("XDG_STATE_HOME", state.path().to_str().unwrap()); + let _config = EnvVarGuard::set("XDG_CONFIG_HOME", config_dir.path().to_str().unwrap()); + + let certificate = rcgen::generate_simple_self_signed(vec!["destination.example".into()]) + .expect("test certificate"); + let source = source_dir.path().join("destination-ca.pem"); + std::fs::write(&source, certificate.cert.pem()).unwrap(); + let config_path = config_dir.path().join("gateway.toml"); + std::fs::write( + &config_path, + format!( + "[openshell]\nversion = 2\n\n[openshell.supervisor.network]\nadditional_ca_bundle = \"{}\"\n", + source.display() + ), + ) + .unwrap(); + + let (mut args, matches) = parse_with_args(&[ + "openshell-gateway", + "--config", + config_path.to_str().unwrap(), + "--db-url", + "sqlite::memory:", + "--disable-tls", + "--compute-driver", + "local", + ]); + let prepared = super::prepare_server_config_with_drivers( + &mut args, + &matches, + &detected_local_registry(), + ) + .expect("global network trust should normalize during startup preparation"); + let bundle = prepared.network_trust.expect("normalized trust bundle"); + assert_eq!(bundle.certificate_count(), 1); + assert!(bundle.artifact_path().is_file()); + assert_eq!(std::fs::read(bundle.artifact_path()).unwrap(), bundle.pem()); + } + + #[test] + fn prepare_server_config_rejects_network_trust_endpoint_override_without_artifact() { + let _lock = ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let state = tempfile::tempdir().unwrap(); + let source_dir = tempfile::tempdir().unwrap(); + let config_dir = tempfile::tempdir().unwrap(); + let _state = EnvVarGuard::set("XDG_STATE_HOME", state.path().to_str().unwrap()); + let _config = EnvVarGuard::set("XDG_CONFIG_HOME", config_dir.path().to_str().unwrap()); + + let certificate = rcgen::generate_simple_self_signed(vec!["destination.example".into()]) + .expect("test certificate"); + let source = source_dir.path().join("destination-ca.pem"); + std::fs::write(&source, certificate.cert.pem()).unwrap(); + let config_path = config_dir.path().join("gateway.toml"); + std::fs::write( + &config_path, + format!( + "[openshell]\nversion = 2\n\n[openshell.supervisor.network]\nadditional_ca_bundle = \"{}\"\n", + source.display() + ), + ) + .unwrap(); + + let (mut args, matches) = parse_with_args(&[ + "openshell-gateway", + "--config", + config_path.to_str().unwrap(), + "--db-url", + "sqlite::memory:", + "--disable-tls", + "--compute-driver", + "local", + "--compute-driver-socket", + "/tmp/local.sock", + ]); + let result = super::prepare_server_config_with_drivers( + &mut args, + &matches, + &detected_local_registry(), + ); + let Err(error) = result else { + panic!("network trust cannot be silently discarded by a remote endpoint"); + }; + + assert!(error.to_string().contains("additional_ca_bundle")); + assert!(error.to_string().contains("unsupported remote endpoint")); + assert!( + !state.path().join("network-supervisor").exists(), + "rejected endpoint override must not stage a trust artifact" + ); + } + #[test] fn registry_detection_drives_auth_defaults_once() { let state = tempfile::tempdir().unwrap(); diff --git a/crates/openshell-server/src/compute/driver_config.rs b/crates/openshell-server/src/compute/driver_config.rs index 12395f4a1..418ae5b10 100644 --- a/crates/openshell-server/src/compute/driver_config.rs +++ b/crates/openshell-server/src/compute/driver_config.rs @@ -9,7 +9,7 @@ use crate::config_file; use crate::defaults::LocalTlsPaths; -use openshell_core::{Error, Result}; +use openshell_core::{Error, NetworkSupervisorTrustBundle, Result}; use serde::Deserialize; use std::collections::BTreeMap; use std::path::PathBuf; @@ -67,6 +67,14 @@ impl GuestTlsPaths { pub struct DriverStartupContext<'a> { pub file: Option<&'a config_file::ConfigFile>, pub guest_tls: Option<&'a GuestTlsPaths>, + /// A staged artifact for runtime driver construction. This is deliberately + /// absent during effective-config preflight. + pub network_trust: Option<&'a NetworkSupervisorTrustBundle>, + /// Whether destination trust sources were configured and normalized. + /// + /// This remains available during preflight so selection can reject a + /// driver that cannot propagate trust, without fabricating an artifact. + pub network_trust_configured: bool, pub gateway_port: u16, pub gateway_tls_enabled: bool, pub endpoint_overrides: &'a BTreeMap, @@ -216,6 +224,8 @@ mod tests { DriverStartupContext { file, guest_tls: None, + network_trust: None, + network_trust_configured: false, gateway_port: openshell_core::config::DEFAULT_SERVER_PORT, gateway_tls_enabled: false, endpoint_overrides, diff --git a/crates/openshell-server/src/compute/mod.rs b/crates/openshell-server/src/compute/mod.rs index 0d2fb40ad..215b864e4 100644 --- a/crates/openshell-server/src/compute/mod.rs +++ b/crates/openshell-server/src/compute/mod.rs @@ -1216,6 +1216,11 @@ impl ComputeRuntime { // Every start path derives resume from the durable transition, so a // retried or recovered start repeats the request that was committed. let resume = in_flight_resume_intent(sandbox); + // Starts may need to replace a stopped local workload/supervisor + // pair. Always reconstruct that input from the gateway's durable + // record; never let a driver derive provisioning input from mutable + // runtime inspection output. + let driver_sandbox = self.driver_sandbox_for_launch(sandbox)?; Ok(StartSandboxRequest { sandbox_id: sandbox.object_id().to_string(), name: sandbox.object_name().to_string(), @@ -1227,6 +1232,7 @@ impl ComputeRuntime { isolation_backend: self.isolation_backend.clone(), resume: resume.is_some(), restart_if_restore_fails: resume.unwrap_or(false), + sandbox: Some(driver_sandbox), }) } @@ -2495,6 +2501,10 @@ impl ComputeRuntime { } else { None }; + // Starts may need to replace a stopped local workload/supervisor pair. + // The request (including its durable provisioning snapshot) is always + // reconstructed from the gateway's record; never let a driver derive + // provisioning input from mutable runtime inspection output. let request = self.driver_start_request(&starting, launch_authentication, expected_runtime_identity)?; let driver = self.driver.clone(); @@ -4034,11 +4044,13 @@ impl ComputeRuntime { /// Should be called once at gateway startup, before watchers spawn, /// so the watch loop sees the post-start state on its first poll. pub async fn start_persisted_sandboxes(&self) -> Result<(), String> { - self.start_persisted_sandboxes_with_authentication( + // The reconciliation futures carry full persisted sandbox records; + // keep them off the caller's stack (clippy::large_futures). + Box::pin(self.start_persisted_sandboxes_with_authentication( |_| async { Ok(Vec::new()) }, |_| async { Ok(()) }, |_| {}, - ) + )) .await } @@ -4057,13 +4069,13 @@ impl ComputeRuntime { authentication_failed: Failed, ) -> Result<(), String> where - Authentication: Fn(&Sandbox) -> AuthenticationFuture, - AuthenticationFuture: Future, String>>, - Committed: Fn(&str) -> CommittedFuture, - CommittedFuture: Future>, - Failed: Fn(&str), + Authentication: Fn(&Sandbox) -> AuthenticationFuture + Send + Sync, + AuthenticationFuture: Future, String>> + Send, + Committed: Fn(&str) -> CommittedFuture + Send + Sync, + CommittedFuture: Future> + Send, + Failed: Fn(&str) + Send + Sync, { - self.recover_persisted_lifecycle_transitions().await?; + Box::pin(self.recover_persisted_lifecycle_transitions()).await?; if !self.driver_info.gateway_manages_lifecycle { return Ok(()); } @@ -4160,6 +4172,9 @@ impl ComputeRuntime { continue; } }; + // Recovery is allowed to reconstruct provisioning inputs only + // from the persisted gateway record (the request snapshot is + // rebuilt by `driver_start_request`). let request = match self.driver_start_request( &sandbox, launch_authentication, @@ -4420,8 +4435,8 @@ impl ComputeRuntime { // Recovery retains the original attempt and deadline. A // stalled driver must release this lifecycle gate after // expiry so the deadline worker can reclaim its compute. - match self - .await_provisioning_operation( + match Box::pin( + self.await_provisioning_operation( &sandbox, self.driver.call_owned( openshell_otel::rpc::START_SANDBOX, @@ -4430,8 +4445,9 @@ impl ComputeRuntime { driver.start_sandbox(Request::new(request)).await }, ), - ) - .await + ), + ) + .await { Ok((_, settled)) => { let updated = self.clear_resume_intent(*settled).await; @@ -4492,6 +4508,11 @@ impl ComputeRuntime { .map_err(|err| format!("failed to list sandboxes for {operation}: {err}")) } + /// Fail closed when a start/recovery cannot reconstruct a driver's durable + /// provisioning snapshot. This is intentionally distinct from an + /// ambiguous transport failure: no start RPC has been issued, so keeping + /// `Starting` for a blind retry would allow a local reconciliation path to + /// fall back to mutable platform state. async fn mark_sandbox_error(&self, sandbox: &Sandbox, reason: &str, message: &str) { let _guard = self.lock_sandbox_local(sandbox.object_id()).await; let sandbox_id = sandbox.object_id().to_string(); @@ -18073,11 +18094,31 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" )); driver.clear_calls(); - runtime.start_persisted_sandboxes().await.unwrap(); + runtime + .start_persisted_sandboxes_with_authentication( + |_| async { Ok(b"test-launch-authentication".to_vec()) }, + |_| async { Ok(()) }, + |_| {}, + ) + .await + .unwrap(); assert!(matches!( driver.calls().as_slice(), - [FakeComputeDriverCall::GetCapabilities, FakeComputeDriverCall::StartSandbox { sandbox_id, sandbox_name, .. }] - if sandbox_id == "sb-uds" && sandbox_name == "uds-sandbox" + [ + FakeComputeDriverCall::GetCapabilities, + FakeComputeDriverCall::StartSandbox { + sandbox_id, + sandbox_name, + launch_authentication, + sandbox: Some(durable), + .. + }, + ] + if sandbox_id == "sb-uds" + && sandbox_name == "uds-sandbox" + && !launch_authentication.is_empty() + && durable.id == "sb-uds" + && durable.name == "uds-sandbox" )); driver.clear_calls(); assert!( @@ -18102,6 +18143,44 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" } } + #[test] + fn start_sandbox_request_uses_optional_wire_field_nine_for_durable_snapshot() { + let legacy = StartSandboxRequest { + sandbox_id: "sandbox-id".into(), + name: "sandbox-name".into(), + launch_authentication: b"opaque-auth".to_vec(), + generation_id: "generation-id".into(), + expected_runtime_identity: String::new(), + isolation_backend: None, + resume: false, + restart_if_restore_fails: false, + sandbox: None, + }; + let with_snapshot = StartSandboxRequest { + sandbox: Some(DriverSandbox { + id: "sandbox-id".into(), + name: "sandbox-name".into(), + ..Default::default() + }), + ..legacy.clone() + }; + let legacy_wire = legacy.encode_to_vec(); + let snapshot_wire = with_snapshot.encode_to_vec(); + // Field 9 is length-delimited, hence its protobuf key is 0x4a. The + // legacy form does not emit it, preserving existing fields 1–8 + // (fields 7 and 8 are the upstream suspend/resume controls). + assert!(!legacy_wire.contains(&0x4a)); + assert!(snapshot_wire.contains(&0x4a)); + assert_eq!( + StartSandboxRequest::decode(snapshot_wire.as_slice()) + .unwrap() + .sandbox + .as_ref() + .map(|sandbox| sandbox.id.as_str()), + Some("sandbox-id") + ); + } + #[tokio::test] #[cfg(unix)] async fn remote_compute_driver_rejects_policy_changes_before_forwarding() { @@ -19992,9 +20071,7 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" runtime.store.put_message(&sandbox).await.unwrap(); let recovered_runtime = runtime.clone(); let mut recovery = tokio::spawn(async move { - recovered_runtime - .recover_persisted_lifecycle_transitions() - .await + Box::pin(recovered_runtime.recover_persisted_lifecycle_transitions()).await }); tokio::time::timeout(Duration::from_secs(1), driver.start_started.notified()) .await diff --git a/crates/openshell-server/src/config_file.rs b/crates/openshell-server/src/config_file.rs index 5085faccd..807b7804b 100644 --- a/crates/openshell-server/src/config_file.rs +++ b/crates/openshell-server/src/config_file.rs @@ -467,6 +467,22 @@ pub struct SupervisorFileSection { /// operator-owned and changes require a gateway restart. #[serde(default)] pub middleware: Vec, + /// Shared network-supervisor configuration. This is intentionally outside + /// the compute-driver tables so every supported driver consumes the same + /// startup contract. + #[serde(default)] + pub network: SupervisorNetworkFileSection, +} + +/// `[openshell.supervisor.network]` configuration. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct SupervisorNetworkFileSection { + /// Gateway-local pre-assembled PEM bundle whose certificates augment the + /// supervisor's destination trust roots. The gateway validates and stages + /// this single file during startup before any compute driver is constructed. + #[serde(default)] + pub additional_ca_bundle: Option, } impl ConfigFile { @@ -864,6 +880,19 @@ fn parse_and_validate(path: &Path, contents: &str) -> Result, pub guest_tls: Option, + pub network_trust: Option, pub compute_driver: ComputeDriverSelection, pub legacy_compute_driver_env_seen: bool, } @@ -599,6 +601,7 @@ pub(crate) async fn run_server( config, config_file, guest_tls, + network_trust, compute_driver, legacy_compute_driver_env_seen: _, } = startup; @@ -715,6 +718,8 @@ pub(crate) async fn run_server( let driver_startup = compute::driver_config::DriverStartupContext { file: config_file.as_ref(), guest_tls: guest_tls.as_ref(), + network_trust: network_trust.as_ref(), + network_trust_configured: network_trust.is_some(), gateway_port: config.bind_address.port(), gateway_tls_enabled: config.tls.is_some(), endpoint_overrides: &config.compute_driver_endpoints, @@ -1441,6 +1446,16 @@ pub trait ComputeDriverFactory: Send + Sync { Ok(Vec::new()) } + /// Whether this factory can propagate the gateway-owned destination trust + /// bundle to its network supervisor. + /// + /// The default is deliberately conservative: out-of-tree, remote, and + /// otherwise unannotated drivers must explicitly opt in before a + /// configured additional CA bundle can be used with them. + fn supports_network_supervisor_trust(&self) -> bool { + false + } + async fn build(&self, context: ComputeDriverBuildContext<'_>) -> Result; } @@ -1788,6 +1803,13 @@ impl ComputeDriverBuildContext<'_> { .map(compute::driver_config::GuestTlsPaths::as_path) } + /// Gateway-normalized destination trust material for the network + /// supervisor. The bundle is absent when the global setting is omitted. + #[must_use] + pub fn network_trust_bundle(&self) -> Option<&NetworkSupervisorTrustBundle> { + self.config.driver_startup.network_trust + } + /// Deserialize the selected driver's merged TOML table. pub fn driver_config(&self) -> Result where @@ -2046,6 +2068,14 @@ fn resolve_configured_compute_driver( ) -> Result { let name = openshell_core::config::normalize_compute_driver_name(driver_name) .map_err(Error::config)?; + let network_trust_configured = + driver_startup.network_trust.is_some() || driver_startup.network_trust_configured; + if network_trust_configured && driver_startup.endpoint_overrides.contains_key(&name) { + return Err(Error::config(format!( + "{} is configured but compute driver '{name}' uses an unsupported remote endpoint", + network_trust::CONFIG_FIELD + ))); + } // An operator-provided endpoint replaces normal construction for the // selected name, including a compiled registration with the same name. // The gateway connects to it; it does not provision the remote driver. @@ -2054,9 +2084,21 @@ fn resolve_configured_compute_driver( } if let Some(registration) = registry.get(&name) { + if network_trust_configured && !registration.factory.supports_network_supervisor_trust() { + return Err(Error::config(format!( + "{} is configured but compute driver '{name}' does not support network trust propagation", + network_trust::CONFIG_FIELD + ))); + } return Ok(ConfiguredComputeDriver::Registered(registration.clone())); } + if network_trust_configured { + return Err(Error::config(format!( + "{} is configured but remote compute driver '{name}' is unsupported because it cannot propagate network trust", + network_trust::CONFIG_FIELD + ))); + } Ok(ConfiguredComputeDriver::Remote { name }) } @@ -2132,7 +2174,7 @@ mod tests { serve_gateway_listener, validate_peer_endpoint_scheme, }; use openshell_core::{ - Config, + Config, NetworkSupervisorTrustBundle, proto::{HealthRequest, open_shell_client::OpenShellClient}, }; use std::io::{Error, ErrorKind}; @@ -2319,16 +2361,35 @@ mod tests { fn test_driver_startup<'a>( config: &'a Config, file: Option<&'a super::config_file::ConfigFile>, + ) -> crate::compute::driver_config::DriverStartupContext<'a> { + test_driver_startup_with_bundle(config, file, None) + } + + fn test_driver_startup_with_bundle<'a>( + config: &'a Config, + file: Option<&'a super::config_file::ConfigFile>, + network_trust: Option<&'a NetworkSupervisorTrustBundle>, ) -> crate::compute::driver_config::DriverStartupContext<'a> { crate::compute::driver_config::DriverStartupContext { file, guest_tls: None, + network_trust, + network_trust_configured: network_trust.is_some(), gateway_port: openshell_core::config::DEFAULT_SERVER_PORT, gateway_tls_enabled: false, endpoint_overrides: &config.compute_driver_endpoints, } } + fn test_network_trust_bundle() -> NetworkSupervisorTrustBundle { + NetworkSupervisorTrustBundle::new( + b"-----BEGIN CERTIFICATE-----\nfixture\n-----END CERTIFICATE-----\n".to_vec(), + 1, + "sha256:fixture", + "/tmp/openshell-additional-ca.crt".into(), + ) + } + fn test_compute_drivers() -> super::ComputeDriverRegistry { let mut registry = super::ComputeDriverRegistry::new(); for (name, priority) in [("alpha", 100), ("beta", 200), ("gamma", 300)] { @@ -2378,6 +2439,8 @@ mod tests { guest_tls: None, gateway_port: 8080, gateway_tls_enabled: true, + network_trust: None, + network_trust_configured: false, endpoint_overrides: &endpoint_overrides, }, }, @@ -2415,6 +2478,8 @@ mod tests { driver_startup: crate::compute::driver_config::DriverStartupContext { file: None, guest_tls: None, + network_trust: None, + network_trust_configured: false, gateway_port: 8080, gateway_tls_enabled: false, endpoint_overrides: &endpoint_overrides, @@ -2981,6 +3046,120 @@ mod tests { )); } + #[test] + fn network_trust_is_available_to_startup_context() { + let config = Config::new(None).with_compute_driver("docker"); + let bundle = test_network_trust_bundle(); + let startup = test_driver_startup_with_bundle(&config, None, Some(&bundle)); + + assert_eq!( + startup + .network_trust + .expect("configured trust bundle") + .certificate_count(), + 1 + ); + } + + #[test] + fn configured_network_trust_rejects_endpoint_override_before_remote_connection() { + let config = Config::new(None) + .with_compute_driver("docker") + .with_compute_driver_endpoint("docker", "/run/openshell/docker.sock"); + let bundle = test_network_trust_bundle(); + let error = configured_compute_driver( + &test_compute_drivers(), + &config, + test_driver_startup_with_bundle(&config, None, Some(&bundle)), + ) + .expect_err("destination trust must not be discarded by a remote override"); + + assert!(error.to_string().contains("additional_ca_bundle")); + assert!(error.to_string().contains("unsupported")); + assert!(!error.to_string().contains("docker.sock")); + } + + #[test] + fn configured_network_trust_rejects_unknown_remote_driver() { + let config = Config::new(None).with_compute_driver("kyma"); + let bundle = test_network_trust_bundle(); + let error = configured_compute_driver( + &test_compute_drivers(), + &config, + test_driver_startup_with_bundle(&config, None, Some(&bundle)), + ) + .expect_err("destination trust must not be silently ignored by a remote driver"); + + assert!(error.to_string().contains("additional_ca_bundle")); + assert!(error.to_string().contains("kyma")); + } + + #[test] + fn configured_network_trust_rejects_unsupported_custom_registration() { + let mut registry = super::ComputeDriverRegistry::new(); + registry + .install( + super::ComputeDriverRegistration::new("custom", 1, None, TestComputeDriverFactory) + .expect("custom registration"), + ) + .expect("install custom registration"); + let config = Config::new(None).with_compute_driver("custom"); + let bundle = test_network_trust_bundle(); + let error = configured_compute_driver( + ®istry, + &config, + test_driver_startup_with_bundle(&config, None, Some(&bundle)), + ) + .expect_err("custom drivers cannot consume the shared trust contract yet"); + + assert!(error.to_string().contains("additional_ca_bundle")); + assert!(error.to_string().contains("custom")); + assert!(error.to_string().contains("network trust propagation")); + } + + #[derive(Clone, Copy)] + struct TrustPropagatingTestFactory; + + #[async_trait::async_trait] + impl super::ComputeDriverFactory for TrustPropagatingTestFactory { + fn supports_network_supervisor_trust(&self) -> bool { + true + } + + async fn build( + &self, + _context: super::ComputeDriverBuildContext<'_>, + ) -> openshell_core::Result { + unreachable!("selection tests do not construct the driver") + } + } + + #[test] + fn configured_network_trust_accepts_explicitly_capable_registration() { + let mut registry = super::ComputeDriverRegistry::new(); + registry + .install( + super::ComputeDriverRegistration::new( + "custom", + 1, + None, + TrustPropagatingTestFactory, + ) + .expect("custom registration"), + ) + .expect("install custom registration"); + let config = Config::new(None).with_compute_driver("custom"); + let bundle = test_network_trust_bundle(); + + let driver = configured_compute_driver( + ®istry, + &config, + test_driver_startup_with_bundle(&config, None, Some(&bundle)), + ) + .expect("an explicitly capable registration accepts destination trust"); + assert!(matches!(driver, ConfiguredComputeDriver::Registered(_))); + } + #[tokio::test] async fn failed_gateway_listener_bind_does_not_attempt_persisted_sandbox_start() { let occupied_listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); diff --git a/crates/openshell-server/src/network_trust.rs b/crates/openshell-server/src/network_trust.rs new file mode 100644 index 000000000..9b6aa2cdc --- /dev/null +++ b/crates/openshell-server/src/network_trust.rs @@ -0,0 +1,1147 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Startup normalization for the network supervisor destination trust bundle. +//! +//! This module is the only gateway-side boundary that reads the operator's +//! configured source paths. Drivers receive the normalized, gateway-owned +//! artifact represented by [`NetworkSupervisorTrustBundle`], never a source +//! path from the TOML file. + +use base64::Engine as _; +use openshell_core::network_trust::MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES; +use openshell_core::{Error, NetworkSupervisorTrustBundle, Result}; +use sha2::{Digest, Sha256}; +use std::fs; +use std::io::{Cursor, ErrorKind, Read, Write}; +use std::path::{Path, PathBuf}; + +use crate::config_file::SupervisorNetworkFileSection; + +pub const CONFIG_FIELD: &str = "openshell.supervisor.network.additional_ca_bundle"; +const ARTIFACT_DIRECTORY: &str = "network-supervisor"; +const ARTIFACT_FILE_PREFIX: &str = "additional-ca-"; +const ARTIFACT_FILE_SUFFIX: &str = ".crt"; +const ARTIFACT_TEMP_FILE_PREFIX: &str = ".additional-ca-"; +const ARTIFACT_TEMP_FILE_SUFFIX: &str = ".tmp"; + +/// Normalized source material that has not yet been staged as a state artifact. +/// +/// Configuration preflight intentionally uses this type rather than a +/// [`NetworkSupervisorTrustBundle`], so it can validate operator-provided +/// sources without creating state directories or files. +pub struct NormalizedNetworkSupervisorTrust { + normalized_pem: Vec, + certificate_count: usize, + digest: String, +} + +impl NormalizedNetworkSupervisorTrust { + pub fn certificate_count(&self) -> usize { + self.certificate_count + } + + pub fn digest(&self) -> &str { + &self.digest + } +} + +/// Read and strictly normalize configured destination roots without writing a +/// gateway state artifact. +/// +/// An absent path preserves the legacy startup behavior and returns `None`. +/// Once configured, the pre-assembled PEM bundle must contain only usable +/// X.509 certificates; no valid subset is accepted when another item is +/// malformed or is a private key. +pub fn normalize_from_config( + config: &SupervisorNetworkFileSection, +) -> Result> { + config + .additional_ca_bundle + .as_deref() + .map(normalize_source_path) + .transpose() +} + +pub fn stage_normalized( + normalized: NormalizedNetworkSupervisorTrust, +) -> Result { + let state_dir = openshell_core::paths::openshell_state_dir().map_err(|error| { + Error::config(format!( + "failed to resolve network trust state directory: {error}" + )) + })?; + stage_normalized_in(normalized, &state_dir) +} + +#[cfg(test)] +fn normalize_and_stage( + source_paths: &[PathBuf], + state_dir: &Path, +) -> Result> { + let Some(source_path) = source_paths.first() else { + return Ok(None); + }; + stage_normalized_in(normalize_source_path(source_path)?, state_dir).map(Some) +} + +fn normalize_source_path(source_path: &Path) -> Result { + if source_path.as_os_str().is_empty() { + return Err(config_error(source_path, "path must not be empty")); + } + let source = read_source_bounded(source_path)?; + let (normalized_pem, certificate_count) = normalize_source(source_path, &source)?; + Ok(NormalizedNetworkSupervisorTrust { + // Deterministic destination-trust generation label, not an + // authenticity proof: trust comes from the gateway's normalization, + // the immutable staged artifact, and TLS verification. Classified as + // a non-security identity hash in openshell-crypto/README.md. + digest: format!("sha256:{:x}", Sha256::digest(&normalized_pem)), + normalized_pem, + certificate_count, + }) +} + +fn read_source_bounded(source_path: &Path) -> Result> { + let path_metadata = fs::symlink_metadata(source_path) + .map_err(|error| config_error(source_path, format_args!("could not be read: {error}")))?; + if !path_metadata.file_type().is_file() { + return Err(config_error(source_path, "is not a regular file")); + } + + let mut options = fs::OpenOptions::new(); + options.read(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + // Avoid blocking on a path replaced with a FIFO after the preflight + // metadata check. Validate the opened handle below to close the race. + options.custom_flags(libc::O_CLOEXEC | libc::O_NOFOLLOW | libc::O_NONBLOCK); + } + let mut file = options + .open(source_path) + .map_err(|error| config_error(source_path, format_args!("could not be read: {error}")))?; + let opened_metadata = file + .metadata() + .map_err(|error| config_error(source_path, format_args!("could not be read: {error}")))?; + if !opened_metadata.file_type().is_file() { + return Err(config_error(source_path, "is not a regular file")); + } + if opened_metadata.len() > MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES as u64 { + return Err(config_error( + source_path, + format_args!( + "exceeds the maximum readable size of {MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES} bytes" + ), + )); + } + + let mut source = Vec::new(); + Read::by_ref(&mut file) + .take((MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES + 1) as u64) + .read_to_end(&mut source) + .map_err(|error| config_error(source_path, format_args!("could not be read: {error}")))?; + if source.len() > MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES { + return Err(config_error( + source_path, + format_args!( + "exceeds the maximum readable size of {MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES} bytes" + ), + )); + } + Ok(source) +} + +fn stage_normalized_in( + normalized: NormalizedNetworkSupervisorTrust, + state_dir: &Path, +) -> Result { + let artifact_path = stage_artifact(state_dir, &normalized.normalized_pem, &normalized.digest)?; + Ok(NetworkSupervisorTrustBundle::new( + normalized.normalized_pem, + normalized.certificate_count, + normalized.digest, + artifact_path, + )) +} + +fn normalize_source(source_path: &Path, source: &[u8]) -> Result<(Vec, usize)> { + // `rustls_pemfile` intentionally skips text outside PEM blocks. The + // gateway contract is stricter: a configured file must not contain a + // valid subset hidden among unrelated or malformed material. + if source + .windows(b"-----BEGIN ".len()) + .any(|window| window == b"-----BEGIN ") + { + validate_pem_envelope(source_path, source)?; + } + + let mut certificates = Vec::new(); + for item in rustls_pemfile::read_all(&mut Cursor::new(source)) { + let item = item.map_err(|error| { + config_error( + source_path, + format_args!("contains malformed PEM data: {error}"), + ) + })?; + let rustls_pemfile::Item::X509Certificate(certificate) = item else { + return Err(config_error( + source_path, + "contains a non-certificate PEM block", + )); + }; + certificates.push(certificate); + } + + if certificates.is_empty() { + return Err(config_error( + source_path, + "contains no PEM certificate blocks", + )); + } + + let mut roots = rustls::RootCertStore::empty(); + let certificate_count = certificates.len(); + let (added, ignored) = roots.add_parsable_certificates(certificates.clone()); + if added != certificate_count || ignored != 0 { + return Err(config_error( + source_path, + "contains a certificate that is not a usable X.509 trust anchor", + )); + } + + let mut normalized = Vec::new(); + for certificate in certificates { + normalized.extend_from_slice(b"-----BEGIN CERTIFICATE-----\n"); + let encoded = base64::engine::general_purpose::STANDARD.encode(certificate.as_ref()); + for line in encoded.as_bytes().chunks(64) { + normalized.extend_from_slice(line); + normalized.push(b'\n'); + } + normalized.extend_from_slice(b"-----END CERTIFICATE-----\n"); + } + Ok((normalized, certificate_count)) +} + +fn validate_pem_envelope(source_path: &Path, source: &[u8]) -> Result<()> { + let text = std::str::from_utf8(source).map_err(|_| { + config_error( + source_path, + "contains malformed PEM data: input is not UTF-8 PEM text", + ) + })?; + let mut begin_label = None; + for line in text.lines() { + let line = line.trim(); + if line.is_empty() { + continue; + } + if let Some(expected_label) = begin_label { + if let Some(end_label) = pem_label(line, "-----END ") { + if end_label != expected_label { + return Err(config_error( + source_path, + "contains malformed PEM data: END label does not match BEGIN label", + )); + } + begin_label = None; + } else if pem_label(line, "-----BEGIN ").is_some() + || !line + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'+' | b'/' | b'=')) + { + return Err(config_error(source_path, "contains malformed PEM data")); + } + } else if let Some(label) = pem_label(line, "-----BEGIN ") { + begin_label = Some(label); + } else { + return Err(config_error( + source_path, + "contains non-PEM content outside certificate blocks", + )); + } + } + if begin_label.is_some() { + return Err(config_error( + source_path, + "contains malformed PEM data: unterminated PEM block", + )); + } + Ok(()) +} + +/// Return a syntactically valid PEM boundary label. Labels are intentionally +/// compared byte-for-byte, preventing a certificate body from being paired +/// with a differently labelled END line. +fn pem_label<'a>(line: &'a str, prefix: &str) -> Option<&'a str> { + let label = line.strip_prefix(prefix)?.strip_suffix("-----")?; + (!label.is_empty() + && label + .bytes() + .all(|byte| byte.is_ascii_uppercase() || byte.is_ascii_digit() || byte == b' ')) + .then_some(label) +} + +fn stage_artifact(state_dir: &Path, normalized: &[u8], digest: &str) -> Result { + let artifact_dir = state_dir.join(ARTIFACT_DIRECTORY); + ensure_artifact_directory(&artifact_dir)?; + + let artifact_path = artifact_dir.join(artifact_file_name(digest)?); + let temporary_path = stage_temporary_artifact(&artifact_dir, normalized)?; + + // Both names are in `artifact_dir`, so hard_link is a same-filesystem, + // no-replace publication operation. Unlike rename/persist, it cannot + // overwrite an existing generation. + let published_new_artifact = match fs::hard_link(&temporary_path, &artifact_path) { + Ok(()) => true, + Err(error) if error.kind() == ErrorKind::AlreadyExists => { + // A concurrent gateway may have published this generation first. + // Reuse it only after the ordinary no-follow, exact-byte, regular + // file, and read-only verification succeeds. + verify_existing_artifact(&artifact_path, normalized, digest)?; + false + } + Err(error) => { + return Err(Error::config(format!( + "failed to atomically publish network supervisor trust artifact '{}': {error}", + artifact_path.display() + ))); + } + }; + + // TempPath provides cleanup on every earlier return path. Drop it before + // syncing a new publication so the directory sync makes both the final + // name and removal of this process's temporary name durable where the + // platform supports directory syncs. + drop(temporary_path); + if published_new_artifact { + sync_artifact_directory(&artifact_dir)?; + } + + // Generations are deliberately never removed or overwritten here. A + // running Docker or Podman sandbox can still be bind-mounting an older + // generation while a later gateway startup stages this one. + Ok(artifact_path) +} + +/// Write a completed, read-only artifact to a private sibling temporary file. +/// +/// `TempPath` retains ownership after its file handle is closed, so it removes +/// the temporary name if publishing or validating a race winner returns an +/// error. +fn stage_temporary_artifact(artifact_dir: &Path, normalized: &[u8]) -> Result { + let mut temporary = tempfile::Builder::new() + .prefix(ARTIFACT_TEMP_FILE_PREFIX) + .suffix(ARTIFACT_TEMP_FILE_SUFFIX) + .tempfile_in(artifact_dir) + .map_err(|error| { + Error::config(format!( + "failed to create network supervisor trust temporary artifact in '{}': {error}", + artifact_dir.display() + )) + })?; + let temporary_path = temporary.path().to_path_buf(); + + temporary.write_all(normalized).map_err(|error| { + Error::config(format!( + "failed to write network supervisor trust temporary artifact '{}': {error}", + temporary_path.display() + )) + })?; + set_artifact_file_permissions(temporary.as_file(), &temporary_path)?; + temporary.as_file().sync_all().map_err(|error| { + Error::config(format!( + "failed to flush network supervisor trust temporary artifact '{}': {error}", + temporary_path.display() + )) + })?; + + // into_temp_path drops the open File before returning while preserving + // TempPath's RAII removal behavior for the sibling temporary name. + Ok(temporary.into_temp_path()) +} + +fn artifact_file_name(digest: &str) -> Result { + let hex = digest.strip_prefix("sha256:").filter(|hex| { + hex.len() == Sha256::output_size() * 2 && hex.bytes().all(|byte| byte.is_ascii_hexdigit()) + }); + let Some(hex) = hex else { + return Err(Error::config( + "failed to derive network supervisor trust artifact name from its digest", + )); + }; + Ok(format!("{ARTIFACT_FILE_PREFIX}{hex}{ARTIFACT_FILE_SUFFIX}")) +} + +fn ensure_artifact_directory(path: &Path) -> Result<()> { + let mut builder = fs::DirBuilder::new(); + builder.recursive(true); + #[cfg(unix)] + { + use std::os::unix::fs::DirBuilderExt as _; + builder.mode(0o700); + } + builder.create(path).map_err(|error| { + Error::config(format!( + "failed to create network supervisor trust artifact directory '{}': {error}", + path.display() + )) + })?; + let metadata = fs::symlink_metadata(path).map_err(|error| { + Error::config(format!( + "failed to inspect network supervisor trust artifact directory '{}': {error}", + path.display() + )) + })?; + if !metadata.file_type().is_dir() { + return Err(Error::config(format!( + "network supervisor trust artifact directory '{}' is not a directory", + path.display() + ))); + } + set_artifact_directory_permissions(path) +} + +fn verify_existing_artifact(path: &Path, normalized: &[u8], digest: &str) -> Result<()> { + NetworkSupervisorTrustBundle::new(normalized.to_vec(), 0, digest, path.to_path_buf()) + .verify_artifact() + .map_err(|error| { + Error::config(format!( + "existing network supervisor trust artifact '{}' cannot be reused: {error}", + path.display() + )) + }) +} + +fn set_artifact_directory_permissions(path: &Path) -> Result<()> { + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + fs::set_permissions(path, fs::Permissions::from_mode(0o700)).map_err(|error| { + Error::config(format!( + "failed to set permissions on network supervisor trust directory '{}': {error}", + path.display() + )) + })?; + } + #[cfg(not(unix))] + let _ = path; + Ok(()) +} + +fn set_artifact_file_permissions(file: &fs::File, path: &Path) -> Result<()> { + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + file.set_permissions(fs::Permissions::from_mode(0o444)) + .map_err(|error| { + Error::config(format!( + "failed to set permissions on network supervisor trust temporary artifact '{}': {error}", + path.display() + )) + })?; + } + #[cfg(not(unix))] + let _ = (file, path); + Ok(()) +} + +/// Durably record a successful hard-link publication on platforms that permit +/// syncing directories. This follows the same `File::open(...).sync_all()` +/// pattern used for other atomic state writes in the repository. +fn sync_artifact_directory(path: &Path) -> Result<()> { + #[cfg(unix)] + { + fs::File::open(path) + .and_then(|directory| directory.sync_all()) + .map_err(|error| { + Error::config(format!( + "failed to sync network supervisor trust artifact directory '{}': {error}", + path.display() + )) + })?; + } + #[cfg(not(unix))] + let _ = path; + Ok(()) +} + +fn config_error(path: &Path, detail: impl std::fmt::Display) -> Error { + Error::config(format!( + "{CONFIG_FIELD} source '{}': {detail}", + path.display() + )) +} + +#[cfg(test)] +mod tests { + use super::*; + use openshell_crypto::pki::generate_simple_self_signed; + use std::fs; + use tempfile::tempdir; + + fn certificate_pem(common_name: &str) -> String { + generate_simple_self_signed(vec![common_name.to_string()]) + .expect("generate test certificate") + .cert + .pem() + } + + fn write_source(dir: &Path, name: &str, contents: impl AsRef<[u8]>) -> PathBuf { + let path = dir.join(name); + fs::write(&path, contents).expect("write source fixture"); + path + } + + fn staged_artifact_path(state_dir: &Path, digest: &str) -> PathBuf { + state_dir + .join(ARTIFACT_DIRECTORY) + .join(artifact_file_name(digest).expect("artifact file name")) + } + + fn normalized_from_source(source: PathBuf) -> NormalizedNetworkSupervisorTrust { + normalize_source_path(&source).expect("normalize source") + } + + fn digest_for(contents: &[u8]) -> String { + format!("sha256:{:x}", Sha256::digest(contents)) + } + + fn assert_no_artifact_temporary_names(state_dir: &Path) { + let artifact_dir = state_dir.join(ARTIFACT_DIRECTORY); + let temporary_names: Vec<_> = fs::read_dir(&artifact_dir) + .expect("read artifact directory") + .map(|entry| entry.expect("read artifact directory entry").file_name()) + .filter(|name| { + name.to_str().is_some_and(|name| { + name.starts_with(ARTIFACT_TEMP_FILE_PREFIX) + && name.ends_with(ARTIFACT_TEMP_FILE_SUFFIX) + }) + }) + .collect(); + assert!( + temporary_names.is_empty(), + "temporary artifacts remained in '{}': {temporary_names:?}", + artifact_dir.display() + ); + } + + fn assert_rejected(path: &Path, state_dir: &Path, expected: &str) { + let error = normalize_and_stage(&[path.to_path_buf()], state_dir) + .expect_err("invalid trust material must fail closed"); + let message = error.to_string(); + assert!(message.contains(CONFIG_FIELD), "missing field in {message}"); + assert!( + message.contains(&path.display().to_string()), + "missing source path in {message}" + ); + assert!( + message.contains(expected), + "missing {expected:?} in {message}" + ); + assert!( + !message.contains("CERTIFICATE"), + "certificate body/framing leaked in {message}" + ); + } + + #[test] + fn normalizes_multiple_certificates_and_stages_redacted_metadata() { + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + let first = certificate_pem("ca-one.example"); + let second = certificate_pem("ca-two.example"); + let source = write_source( + source_dir.path(), + "bundle.pem", + format!("{first}\n{second}"), + ); + + let bundle = normalize_and_stage(&[source], state_dir.path()) + .expect("normalize") + .expect("configured bundle"); + assert_eq!(bundle.certificate_count(), 2); + assert!(bundle.digest().starts_with("sha256:")); + assert_eq!( + fs::read(bundle.artifact_path()).expect("artifact"), + bundle.pem() + ); + let digest_hex = bundle + .digest() + .strip_prefix("sha256:") + .expect("digest prefix"); + assert_eq!(digest_hex.len(), Sha256::output_size() * 2); + assert_eq!( + bundle.artifact_path(), + state_dir + .path() + .join(ARTIFACT_DIRECTORY) + .join(format!("additional-ca-{digest_hex}.crt")) + ); + let debug = format!("{bundle:?}"); + assert!(!debug.contains(&first)); + assert!(!debug.contains(&second)); + } + + #[test] + fn same_generation_reuses_the_content_addressed_artifact() { + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + let source = write_source( + source_dir.path(), + "bundle.pem", + certificate_pem("same-generation.example"), + ); + + let first = normalize_and_stage(std::slice::from_ref(&source), state_dir.path()) + .expect("first stage") + .expect("configured bundle"); + let second = normalize_and_stage(&[source], state_dir.path()) + .expect("second stage") + .expect("configured bundle"); + + assert_eq!(first.digest(), second.digest()); + assert_eq!(first.artifact_path(), second.artifact_path()); + assert_eq!( + fs::read(second.artifact_path()).expect("artifact"), + second.pem() + ); + } + + #[test] + fn successful_publication_removes_its_temporary_artifact() { + let state_dir = tempdir().expect("state dir"); + let normalized = b"canonical normalized CA bytes\n"; + let digest = digest_for(normalized); + + let artifact_path = stage_artifact(state_dir.path(), normalized, &digest) + .expect("publish a new artifact generation"); + + assert_eq!( + artifact_path, + staged_artifact_path(state_dir.path(), &digest) + ); + assert_eq!(fs::read(&artifact_path).expect("artifact"), normalized); + verify_existing_artifact(&artifact_path, normalized, &digest) + .expect("new artifact satisfies reuse validation"); + assert_no_artifact_temporary_names(state_dir.path()); + } + + #[test] + fn existing_race_winner_is_validated_and_removes_its_temporary_artifact() { + let state_dir = tempdir().expect("state dir"); + let normalized = b"canonical normalized CA bytes\n"; + let digest = digest_for(normalized); + + let first_path = + stage_artifact(state_dir.path(), normalized, &digest).expect("publish the race winner"); + let reused_path = stage_artifact(state_dir.path(), normalized, &digest) + .expect("validate and reuse existing race winner"); + + assert_eq!(reused_path, first_path); + assert_eq!(fs::read(&reused_path).expect("artifact"), normalized); + assert_no_artifact_temporary_names(state_dir.path()); + } + + #[test] + fn concurrent_same_generation_publication_returns_one_exact_artifact() { + use std::sync::{Arc, Barrier}; + use std::thread; + + const CONTENDERS: usize = 12; + + let state_dir = tempdir().expect("state dir"); + let normalized = Arc::new(b"canonical normalized CA bytes\n".to_vec()); + let digest = Arc::new(digest_for(&normalized)); + let expected_path = staged_artifact_path(state_dir.path(), &digest); + let barrier = Arc::new(Barrier::new(CONTENDERS)); + // Spawn every contender before joining: each thread waits on the same + // barrier, so joining while constructing this list would deadlock. + let mut contenders = Vec::with_capacity(CONTENDERS); + for _ in 0..CONTENDERS { + let state_dir = state_dir.path().to_path_buf(); + let normalized = Arc::clone(&normalized); + let digest = Arc::clone(&digest); + let barrier = Arc::clone(&barrier); + contenders.push(thread::spawn(move || { + barrier.wait(); + stage_artifact(&state_dir, normalized.as_slice(), digest.as_str()) + .map_err(|error| error.to_string()) + })); + } + + let paths: Vec<_> = contenders + .into_iter() + .map(|contender| { + contender + .join() + .expect("publication contender must not panic") + .expect("each contender must publish or validate the same generation") + }) + .collect(); + assert!( + paths.iter().all(|path| path == &expected_path), + "all contenders must return the same artifact path: {paths:?}" + ); + assert_eq!( + fs::read(&expected_path).expect("artifact"), + normalized.as_slice() + ); + verify_existing_artifact(&expected_path, normalized.as_slice(), &digest) + .expect("concurrent publication leaves a valid artifact"); + assert_no_artifact_temporary_names(state_dir.path()); + } + + #[test] + fn failed_existing_generation_validation_cleans_its_temporary_artifact() { + let state_dir = tempdir().expect("state dir"); + let normalized = b"canonical normalized CA bytes\n"; + let digest = digest_for(normalized); + let artifact_path = staged_artifact_path(state_dir.path(), &digest); + ensure_artifact_directory(artifact_path.parent().expect("artifact directory")) + .expect("create artifact directory"); + let replacement = b"different CA bytes that must not be disclosed\n"; + fs::write(&artifact_path, replacement).expect("write invalid existing generation"); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + fs::set_permissions(&artifact_path, fs::Permissions::from_mode(0o444)) + .expect("make invalid generation read-only"); + } + + let error = stage_artifact(state_dir.path(), normalized, &digest) + .expect_err("a wrong existing generation must fail closed"); + + assert!(error.to_string().contains("does not match")); + assert!(!error.to_string().contains("different CA bytes")); + assert_eq!(fs::read(&artifact_path).expect("artifact"), replacement); + assert_no_artifact_temporary_names(state_dir.path()); + } + + #[test] + fn staging_does_not_remove_preexisting_temporary_names() { + let state_dir = tempdir().expect("state dir"); + let artifact_dir = state_dir.path().join(ARTIFACT_DIRECTORY); + ensure_artifact_directory(&artifact_dir).expect("create artifact directory"); + let stale_temporary = artifact_dir.join(".additional-ca-stale.tmp"); + fs::write(&stale_temporary, b"stale temporary bytes").expect("write stale temporary"); + let normalized = b"canonical normalized CA bytes\n"; + let digest = digest_for(normalized); + + stage_artifact(state_dir.path(), normalized, &digest).expect("publish artifact"); + + assert_eq!( + fs::read(&stale_temporary).expect("stale temporary"), + b"stale temporary bytes" + ); + } + + #[cfg(unix)] + #[test] + fn artifact_directory_sync_reports_a_missing_directory() { + let state_dir = tempdir().expect("state dir"); + let missing = state_dir.path().join("missing-artifact-directory"); + + let error = sync_artifact_directory(&missing) + .expect_err("syncing a missing artifact directory must fail"); + + assert!(error.to_string().contains(&missing.display().to_string())); + } + + #[test] + fn different_generations_have_distinct_content_addressed_artifacts() { + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + let first_source = write_source( + source_dir.path(), + "first.pem", + certificate_pem("first-generation.example"), + ); + let second_source = write_source( + source_dir.path(), + "second.pem", + certificate_pem("second-generation.example"), + ); + + let first = normalize_and_stage(&[first_source], state_dir.path()) + .expect("first stage") + .expect("configured bundle"); + let second = normalize_and_stage(&[second_source], state_dir.path()) + .expect("second stage") + .expect("configured bundle"); + + assert_ne!(first.digest(), second.digest()); + assert_ne!(first.artifact_path(), second.artifact_path()); + assert!(first.artifact_path().is_file()); + assert!(second.artifact_path().is_file()); + } + + #[test] + fn fixed_legacy_artifact_path_is_not_consumed() { + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + let legacy_dir = state_dir.path().join(ARTIFACT_DIRECTORY); + fs::create_dir(&legacy_dir).expect("legacy artifact directory"); + let legacy_path = legacy_dir.join("additional-ca.crt"); + let legacy_contents = b"legacy artifact must remain untouched"; + fs::write(&legacy_path, legacy_contents).expect("legacy artifact"); + let source = write_source( + source_dir.path(), + "bundle.pem", + certificate_pem("legacy-path.example"), + ); + + let bundle = normalize_and_stage(&[source], state_dir.path()) + .expect("stage") + .expect("configured bundle"); + + assert_ne!(bundle.artifact_path(), legacy_path); + assert_eq!( + fs::read(&legacy_path).expect("legacy artifact"), + legacy_contents + ); + } + + #[cfg(unix)] + #[test] + fn staged_artifact_and_its_directory_have_restrictive_permissions() { + use std::os::unix::fs::PermissionsExt as _; + + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + let source = write_source( + source_dir.path(), + "bundle.pem", + certificate_pem("permissions.example"), + ); + let bundle = normalize_and_stage(&[source], state_dir.path()) + .expect("stage") + .expect("configured bundle"); + + assert_eq!( + fs::metadata(state_dir.path().join(ARTIFACT_DIRECTORY)) + .expect("artifact directory") + .permissions() + .mode() + & 0o7777, + 0o700 + ); + assert_eq!( + fs::metadata(bundle.artifact_path()) + .expect("artifact") + .permissions() + .mode() + & 0o7777, + 0o444 + ); + } + + #[cfg(unix)] + #[test] + fn rejects_preexisting_symlink_generation_without_changing_it() { + use std::os::unix::fs::symlink; + + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + let source = write_source( + source_dir.path(), + "bundle.pem", + certificate_pem("symlink-generation.example"), + ); + let normalized = normalized_from_source(source); + let artifact_path = staged_artifact_path(state_dir.path(), normalized.digest()); + fs::create_dir(artifact_path.parent().expect("artifact parent")).expect("artifact dir"); + let target = state_dir.path().join("symlink-target"); + let target_contents = b"symlink target bytes must not leak"; + fs::write(&target, target_contents).expect("target"); + symlink(&target, &artifact_path).expect("artifact symlink"); + + let error = stage_normalized_in(normalized, state_dir.path()) + .expect_err("symlink generation must be rejected"); + assert!(error.to_string().contains("not a regular file")); + assert!(!error.to_string().contains("symlink target bytes")); + assert!( + fs::symlink_metadata(&artifact_path) + .expect("artifact link") + .file_type() + .is_symlink() + ); + assert_eq!(fs::read(target).expect("target"), target_contents); + } + + #[test] + fn rejects_preexisting_directory_generation_without_changing_it() { + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + let source = write_source( + source_dir.path(), + "bundle.pem", + certificate_pem("directory-generation.example"), + ); + let normalized = normalized_from_source(source); + let artifact_path = staged_artifact_path(state_dir.path(), normalized.digest()); + fs::create_dir_all(&artifact_path).expect("artifact directory"); + + let error = stage_normalized_in(normalized, state_dir.path()) + .expect_err("directory generation must be rejected"); + assert!(error.to_string().contains("not a regular file")); + assert!( + fs::metadata(&artifact_path) + .expect("artifact directory") + .is_dir() + ); + } + + #[cfg(unix)] + #[test] + fn rejects_preexisting_wrong_generation_bytes_without_changing_or_disclosing_them() { + use std::os::unix::fs::PermissionsExt as _; + + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + let source = write_source( + source_dir.path(), + "bundle.pem", + certificate_pem("wrong-bytes-generation.example"), + ); + let normalized = normalized_from_source(source); + let artifact_path = staged_artifact_path(state_dir.path(), normalized.digest()); + fs::create_dir(artifact_path.parent().expect("artifact parent")).expect("artifact dir"); + let replacement = b"replacement CA bytes must not leak"; + fs::write(&artifact_path, replacement).expect("replacement artifact"); + fs::set_permissions(&artifact_path, fs::Permissions::from_mode(0o444)) + .expect("read-only replacement"); + + let error = stage_normalized_in(normalized, state_dir.path()) + .expect_err("wrong generation bytes must be rejected"); + assert!(error.to_string().contains("does not match")); + assert!(!error.to_string().contains("replacement CA bytes")); + assert_eq!(fs::read(&artifact_path).expect("artifact"), replacement); + } + + #[cfg(unix)] + #[test] + fn rejects_preexisting_oversized_generation_without_changing_it() { + use std::os::unix::fs::PermissionsExt as _; + + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + let source = write_source( + source_dir.path(), + "bundle.pem", + certificate_pem("oversized-generation.example"), + ); + let normalized = normalized_from_source(source); + let artifact_path = staged_artifact_path(state_dir.path(), normalized.digest()); + fs::create_dir(artifact_path.parent().expect("artifact parent")).expect("artifact dir"); + let oversized = vec![b'x'; MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES + 1]; + fs::write(&artifact_path, &oversized).expect("oversized artifact"); + fs::set_permissions(&artifact_path, fs::Permissions::from_mode(0o444)) + .expect("read-only permissions"); + + let error = stage_normalized_in(normalized, state_dir.path()) + .expect_err("oversized generation must be rejected"); + assert!(error.to_string().contains("exceeds the shared")); + assert!(!error.to_string().contains(&"x".repeat(64))); + assert_eq!( + fs::metadata(&artifact_path).expect("artifact").len(), + oversized.len() as u64 + ); + } + + #[cfg(unix)] + #[test] + fn rejects_preexisting_writable_generation_without_changing_or_disclosing_it() { + use std::os::unix::fs::PermissionsExt as _; + + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + let source = write_source( + source_dir.path(), + "bundle.pem", + certificate_pem("writable-generation.example"), + ); + let normalized = normalized_from_source(source); + let artifact_path = staged_artifact_path(state_dir.path(), normalized.digest()); + fs::create_dir(artifact_path.parent().expect("artifact parent")).expect("artifact dir"); + let expected_bytes = normalized.normalized_pem.clone(); + fs::write(&artifact_path, &expected_bytes).expect("writable artifact"); + fs::set_permissions(&artifact_path, fs::Permissions::from_mode(0o644)) + .expect("writable permissions"); + + let error = stage_normalized_in(normalized, state_dir.path()) + .expect_err("writable generation must be rejected"); + assert!(error.to_string().contains("required read-only permissions")); + assert!(!error.to_string().contains("BEGIN CERTIFICATE")); + assert_eq!(fs::read(&artifact_path).expect("artifact"), expected_bytes); + assert_eq!( + fs::metadata(&artifact_path) + .expect("artifact") + .permissions() + .mode() + & 0o7777, + 0o644 + ); + } + + #[test] + fn omitted_sources_do_not_create_an_artifact() { + let state_dir = tempdir().expect("state dir"); + assert!( + normalize_and_stage(&[], state_dir.path()) + .expect("no setting") + .is_none() + ); + assert!(!state_dir.path().join(ARTIFACT_DIRECTORY).exists()); + } + + #[test] + fn rejects_missing_source() { + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + assert_rejected( + &source_dir.path().join("missing.pem"), + state_dir.path(), + "could not be read", + ); + } + + #[test] + fn rejects_empty_source() { + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + let source = write_source(source_dir.path(), "empty.pem", []); + assert_rejected(&source, state_dir.path(), "no PEM certificate blocks"); + } + + #[test] + fn rejects_malformed_source() { + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + let source = write_source(source_dir.path(), "malformed.pem", b"not pem"); + assert_rejected(&source, state_dir.path(), "no PEM certificate blocks"); + } + + #[test] + fn rejects_private_key_source() { + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + let generated = + generate_simple_self_signed(vec!["key.example".to_string()]).expect("certificate"); + let source = write_source( + source_dir.path(), + "private-key.pem", + generated.key_pair.serialize_pem().unwrap(), + ); + assert_rejected(&source, state_dir.path(), "non-certificate PEM block"); + } + + #[test] + fn rejects_mixed_certificate_and_private_key_source() { + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + let generated = + generate_simple_self_signed(vec!["mixed.example".to_string()]).expect("certificate"); + let source = write_source( + source_dir.path(), + "mixed.pem", + format!( + "{}\n{}", + generated.cert.pem(), + generated.key_pair.serialize_pem().unwrap() + ), + ); + assert_rejected(&source, state_dir.path(), "non-certificate PEM block"); + } + + #[test] + fn rejects_valid_certificate_mixed_with_malformed_text() { + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + let certificate = certificate_pem("malformed-mix.example"); + let source = write_source( + source_dir.path(), + "malformed-mix.pem", + format!("{certificate}\nnot pem content"), + ); + assert_rejected( + &source, + state_dir.path(), + "non-PEM content outside certificate blocks", + ); + } + + #[test] + fn rejects_empty_path_entry() { + let state_dir = tempdir().expect("state dir"); + assert_rejected(Path::new(""), state_dir.path(), "path must not be empty"); + } + + #[test] + fn rejects_certificate_free_source_without_leaking_fixture_body() { + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + let fixture_body = "-----BEGIN PRIVATE KEY-----\nfixture-body\n-----END PRIVATE KEY-----\n"; + let source = write_source(source_dir.path(), "non-certificate.pem", fixture_body); + let error = normalize_and_stage(std::slice::from_ref(&source), state_dir.path()) + .expect_err("non-certificate source must fail"); + let message = error.to_string(); + assert!(message.contains(CONFIG_FIELD)); + assert!(message.contains(&source.display().to_string())); + assert!(!message.contains(fixture_body)); + } + + #[cfg(unix)] + #[test] + fn rejects_fifo_and_device_sources_without_blocking() { + use nix::sys::stat::Mode; + use nix::unistd::mkfifo; + + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + let fifo = source_dir.path().join("destination-ca.fifo"); + mkfifo(&fifo, Mode::S_IRUSR | Mode::S_IWUSR).expect("create FIFO"); + + assert_rejected(&fifo, state_dir.path(), "is not a regular file"); + assert_rejected( + Path::new("/dev/zero"), + state_dir.path(), + "is not a regular file", + ); + } + + #[test] + fn rejects_over_limit_source_reads_without_leaking_source_contents() { + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + let source = write_source( + source_dir.path(), + "oversized.pem", + vec![b's'; MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES + 1], + ); + let error = normalize_and_stage(std::slice::from_ref(&source), state_dir.path()) + .expect_err("source reads must be bounded"); + let message = error.to_string(); + assert!(message.contains(CONFIG_FIELD)); + assert!(message.contains("maximum readable size")); + assert!(message.contains(&MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES.to_string())); + assert!(!message.contains(&"s".repeat(32))); + assert!(!state_dir.path().join(ARTIFACT_DIRECTORY).exists()); + } + + #[test] + fn rejects_mismatched_pem_boundary_labels() { + let source_dir = tempdir().expect("source dir"); + let state_dir = tempdir().expect("state dir"); + let source = write_source( + source_dir.path(), + "mismatched-label.pem", + "-----BEGIN CERTIFICATE-----\nZmFrZQ==\n-----END PRIVATE KEY-----\n", + ); + assert_rejected( + &source, + state_dir.path(), + "END label does not match BEGIN label", + ); + } +} diff --git a/crates/openshell-server/src/test_support.rs b/crates/openshell-server/src/test_support.rs index 112b47cf4..454af49d8 100644 --- a/crates/openshell-server/src/test_support.rs +++ b/crates/openshell-server/src/test_support.rs @@ -113,7 +113,14 @@ pub enum FakeComputeDriverCall { sandbox_id: String, sandbox_name: String, isolation_backend: Option, + /// Fresh supervisor-facing credentials. These remain opaque in test + /// diagnostics just as they do in production tracing. launch_authentication: Vec, + generation_id: String, + /// The optional durable provisioning snapshot supplied by newer + /// gateways. Keeping it in the fake makes compatibility and + /// reconciliation tests able to verify the start contract. + sandbox: Option, }, DeleteSandbox { sandbox_id: String, @@ -437,6 +444,8 @@ impl ComputeDriver for FakeComputeDriver { sandbox_name: request.name, isolation_backend: request.isolation_backend, launch_authentication: request.launch_authentication, + generation_id: request.generation_id, + sandbox: request.sandbox, }); }); Ok(Response::new(StartSandboxResponse::default())) diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index c68b2b1d9..4ea8684f6 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -211,6 +211,77 @@ See [`values.yaml`](values.yaml) for source defaults. Selected overlays: - [`ci/values-spire.yaml`](ci/values-spire.yaml) - SPIFFE/SPIRE provider token grants - [`ci/values-spire-stack.yaml`](ci/values-spire-stack.yaml) - SPIRE hardened chart values for local development +### Additional destination CAs + +Create an operator-owned ConfigMap with a `ca.crt` key when sandboxes must reach +TLS destinations signed by a private CA: + +```shell +kubectl create configmap private-destination-ca -n openshell \ + --from-file=ca.crt=/path/to/ca.crt +``` + +Then set the source ConfigMap name: + +```yaml +supervisor: + network: + additionalCaConfigMapName: private-destination-ca +``` + +The chart mounts the source into the gateway and renders +`[openshell.supervisor.network].additional_ca_bundle`. The gateway validates +and normalizes the one certificate-only PEM bundle at startup. The Kubernetes +driver creates a separate managed ConfigMap in each target sandbox namespace +and mounts it only in the dedicated supervisor Pod at +`/etc/openshell/tls/additional/ca-bundle.pem`. That supervisor receives the +path through `--additional-ca-bundle` and folds the bundle into system roots. +The configured roots augment default destination trust. They do not configure +the corporate proxy, OIDC, or gateway callback mTLS trust. + +Changing only the source ConfigMap does not roll or reload the gateway. Restart +the gateway, then recreate or restart affected sandboxes to use the new roots. +Removing `additionalCaConfigMapName` removes the global TOML setting; newly +created or recreated sandbox pods then have no additional-CA volume or mount. +Existing running sandbox supervisors retain their startup trust until they are +restarted or recreated. + +The driver creates one immutable, content-addressed ConfigMap generation in +each target namespace. Names begin with +`openshell-additional-ca-` and include DNS-safe hashes of the effective +gateway ID and normalized bundle generation. Existing objects are accepted only +when their ownership labels, generation annotation, immutability, and exact +`ca.crt` data match; the driver never patches or updates them. + +Kubernetes cannot restrict `create` by `resourceNames`, and `resourceNames` +cannot express the digest-bearing name prefix needed for `get`. Enabling this +feature therefore gives the gateway/driver service account namespace-wide +ConfigMap `get` and `create` permissions in shared mode. In managed/operator +modes, the chart's ClusterRoleBinding makes those permissions cluster-wide for +the bound service account. The chart does not grant ConfigMap `list`, `watch`, +`patch`, `update`, or `delete`. Use a dedicated gateway service account and +namespace-scoped bindings when this boundary matters. If you render the TOML +setting outside this Helm value, grant `get` and `create` in every target +sandbox namespace. + +Disabling the setting intentionally does not delete driver-managed generations: +the gateway has no ConfigMap list/delete permissions, and an older object may +still be referenced by a running or stopped sandbox. An operator with separate +list/delete authority can inventory the gateway's generations by label: + +```shell +kubectl -n get configmap \ + -l 'openshell.ai/managed-by=openshell,openshell.ai/gateway-id=' +``` + +Before deleting a named generation, verify that no Sandbox resource or pod +volume in that namespace references it. Delete only the confirmed unreferenced +name; removing a referenced generation can make a later pod start or reschedule +fail closed. + +A configured CA can authenticate every policy-permitted TLS endpoint with a +matching certificate chain and hostname. Scope private CAs accordingly. + ### Database backend By default, OpenShell uses SQLite and runs the gateway as a StatefulSet so the @@ -626,6 +697,7 @@ discovery endpoint or its TLS CA. | supervisor.image.registry | string | `""` | Supervisor image registry. Empty uses global.image.registry. | | supervisor.image.repository | string | `"openshell/supervisor"` | Supervisor image repository. | | supervisor.image.tag | string | `""` | Supervisor image tag. Defaults to the chart appVersion when empty. | +| supervisor.network.additionalCaConfigMapName | string | `""` | Name of an operator-owned ConfigMap with a `ca.crt` key containing additional destination CA certificates for sandbox supervisors. Empty disables additional destination trust. | | supervisor.podDisruptionBudget.enabled | bool | `true` | Render a PodDisruptionBudget with maxUnavailable 1 for the shared supervisor, used only when gatewayConfig selects supervisor_topology = "gateway". Disable it only when maintenance must evict more than one supervisor Pod at a time, or evict a member while another is unavailable. | | supervisor.podSecurityContext.fsGroup | int | `65534` | fsGroup assigned to the shared supervisor pod. | | supervisor.podSecurityContext.runAsGroup | int | `65534` | Primary GID assigned to the shared supervisor pod. | diff --git a/deploy/helm/openshell/README.md.gotmpl b/deploy/helm/openshell/README.md.gotmpl index 6eccc800d..479ead340 100644 --- a/deploy/helm/openshell/README.md.gotmpl +++ b/deploy/helm/openshell/README.md.gotmpl @@ -212,6 +212,77 @@ See [`values.yaml`](values.yaml) for source defaults. Selected overlays: - [`ci/values-spire.yaml`](ci/values-spire.yaml) - SPIFFE/SPIRE provider token grants - [`ci/values-spire-stack.yaml`](ci/values-spire-stack.yaml) - SPIRE hardened chart values for local development +### Additional destination CAs + +Create an operator-owned ConfigMap with a `ca.crt` key when sandboxes must reach +TLS destinations signed by a private CA: + +```shell +kubectl create configmap private-destination-ca -n openshell \ + --from-file=ca.crt=/path/to/ca.crt +``` + +Then set the source ConfigMap name: + +```yaml +supervisor: + network: + additionalCaConfigMapName: private-destination-ca +``` + +The chart mounts the source into the gateway and renders +`[openshell.supervisor.network].additional_ca_bundle`. The gateway validates +and normalizes the one certificate-only PEM bundle at startup. The Kubernetes +driver creates a separate managed ConfigMap in each target sandbox namespace +and mounts it only in the dedicated supervisor Pod at +`/etc/openshell/tls/additional/ca-bundle.pem`. That supervisor receives the +path through `--additional-ca-bundle` and folds the bundle into system roots. +The configured roots augment default destination trust. They do not configure +the corporate proxy, OIDC, or gateway callback mTLS trust. + +Changing only the source ConfigMap does not roll or reload the gateway. Restart +the gateway, then recreate or restart affected sandboxes to use the new roots. +Removing `additionalCaConfigMapName` removes the global TOML setting; newly +created or recreated sandbox pods then have no additional-CA volume or mount. +Existing running sandbox supervisors retain their startup trust until they are +restarted or recreated. + +The driver creates one immutable, content-addressed ConfigMap generation in +each target namespace. Names begin with +`openshell-additional-ca-` and include DNS-safe hashes of the effective +gateway ID and normalized bundle generation. Existing objects are accepted only +when their ownership labels, generation annotation, immutability, and exact +`ca.crt` data match; the driver never patches or updates them. + +Kubernetes cannot restrict `create` by `resourceNames`, and `resourceNames` +cannot express the digest-bearing name prefix needed for `get`. Enabling this +feature therefore gives the gateway/driver service account namespace-wide +ConfigMap `get` and `create` permissions in shared mode. In managed/operator +modes, the chart's ClusterRoleBinding makes those permissions cluster-wide for +the bound service account. The chart does not grant ConfigMap `list`, `watch`, +`patch`, `update`, or `delete`. Use a dedicated gateway service account and +namespace-scoped bindings when this boundary matters. If you render the TOML +setting outside this Helm value, grant `get` and `create` in every target +sandbox namespace. + +Disabling the setting intentionally does not delete driver-managed generations: +the gateway has no ConfigMap list/delete permissions, and an older object may +still be referenced by a running or stopped sandbox. An operator with separate +list/delete authority can inventory the gateway's generations by label: + +```shell +kubectl -n get configmap \ + -l 'openshell.ai/managed-by=openshell,openshell.ai/gateway-id=' +``` + +Before deleting a named generation, verify that no Sandbox resource or pod +volume in that namespace references it. Delete only the confirmed unreferenced +name; removing a referenced generation can make a later pod start or reschedule +fail closed. + +A configured CA can authenticate every policy-permitted TLS endpoint with a +matching certificate chain and hostname. Scope private CAs accordingly. + ### Database backend By default, OpenShell uses SQLite and runs the gateway as a StatefulSet so the diff --git a/deploy/helm/openshell/templates/_gateway-workload.tpl b/deploy/helm/openshell/templates/_gateway-workload.tpl index 860be9421..a727b0af8 100644 --- a/deploy/helm/openshell/templates/_gateway-workload.tpl +++ b/deploy/helm/openshell/templates/_gateway-workload.tpl @@ -17,12 +17,14 @@ Gateway pod template shared by the StatefulSet and Deployment workload shapes. {{- $legacyCredentialDrivers := .Values.server.credentialDrivers | default dict -}} {{- $legacyVaultResources := get $legacyCredentialDrivers "vault" | default dict -}} {{- $vaultCaConfigMapName := get $vaultResources "caConfigMapName" | default (get $legacyVaultResources "caConfigMapName") -}} +{{- $supervisorNetwork := .Values.supervisor.network | default dict -}} metadata: annotations: # Roll the gateway workload when the rendered gateway TOML changes - the # gateway only reads /etc/openshell/gateway.toml at startup, so without # this annotation a `helm upgrade` that only mutates the ConfigMap would - # leave pods running with stale config. + # leave pods running with stale config. The operator-owned additional-CA + # ConfigMap is not checksummed; restart the gateway after changing its data. checksum/gateway-config: {{ include (print $.Template.BasePath "/gateway-config.yaml") . | sha256sum }} {{- with .Values.podAnnotations }} {{- toYaml . | nindent 4 }} @@ -134,6 +136,11 @@ spec: - name: SSL_CERT_FILE value: /etc/openshell-tls/oidc-ca/ca.crt {{- end }} + # The gateway stages normalized, content-addressed supervisor trust + # artifacts under its XDG state directory. Keep that state on a + # dedicated writable volume for both StatefulSet and Deployment modes. + - name: XDG_STATE_HOME + value: /var/lib/openshell-state - name: OPENSHELL_TELEMETRY_ENABLED value: {{ .Values.server.telemetryEnabled | quote }} {{- if $spiffeSocketPath }} @@ -161,6 +168,12 @@ spec: - name: gateway-peer-token mountPath: /var/run/secrets/openshell-peer readOnly: true + {{- if get $supervisorNetwork "additionalCaConfigMapName" }} + - name: additional-ca-source + mountPath: /etc/openshell-tls/additional-ca-source/ca.crt + subPath: ca.crt + readOnly: true + {{- end }} {{- if not .Values.server.disableTls }} - name: tls-cert mountPath: /etc/openshell-tls/server @@ -202,6 +215,8 @@ spec: {{- with .Values.server.extraVolumeMounts }} {{- toYaml . | nindent 8 }} {{- end }} + - name: gateway-state + mountPath: /var/lib/openshell-state ports: - name: grpc containerPort: {{ .Values.service.port }} @@ -255,6 +270,14 @@ spec: path: token audience: openshell-gateway-peer expirationSeconds: 3600 + {{- if get $supervisorNetwork "additionalCaConfigMapName" }} + - name: additional-ca-source + configMap: + name: {{ get $supervisorNetwork "additionalCaConfigMapName" | quote }} + items: + - key: ca.crt + path: ca.crt + {{- end }} {{- if not .Values.server.disableTls }} - name: tls-cert secret: @@ -310,6 +333,8 @@ spec: {{- with .Values.server.extraVolumes }} {{- toYaml . | nindent 4 }} {{- end }} + - name: gateway-state + emptyDir: {} {{- with .Values.nodeSelector }} nodeSelector: {{- toYaml . | nindent 4 }} diff --git a/deploy/helm/openshell/templates/_toml.tpl b/deploy/helm/openshell/templates/_toml.tpl index 7f1a0bdfa..cf8bf6120 100644 --- a/deploy/helm/openshell/templates/_toml.tpl +++ b/deploy/helm/openshell/templates/_toml.tpl @@ -315,6 +315,16 @@ openshell.supervisorPoolConfig. */}} {{- if $supervisorPool -}} {{- $_ := set $config "openshell.gateway.supervisor_pool" $supervisorPool -}} {{- end -}} +{{/* Additional destination CA trust is chart plumbing: the runtime table is +owned by the chart, not gatewayConfig. Derive it only from the chart-owned +operator ConfigMap reference. */}} +{{- $supervisorNetwork := .Values.supervisor.network | default dict -}} +{{- if get $supervisorNetwork "additionalCaConfigMapName" -}} +{{- if hasKey $config "openshell.supervisor.network" -}} +{{- fail "gatewayConfig must not define openshell.supervisor.network; the chart derives it from supervisor.network.additionalCaConfigMapName" -}} +{{- end -}} +{{- $_ := set $config "openshell.supervisor.network" (dict "additional_ca_bundle" "/etc/openshell-tls/additional-ca-source/ca.crt") -}} +{{- end -}} {{/* A nested YAML map and its dotted top-level table would serialize to conflicting TOML definitions. Fail before rendering an invalid ConfigMap. */}} {{- range $tableName := keys $config | sortAlpha -}} diff --git a/deploy/helm/openshell/templates/clusterrole.yaml b/deploy/helm/openshell/templates/clusterrole.yaml index a8f2e6f57..8af91e190 100644 --- a/deploy/helm/openshell/templates/clusterrole.yaml +++ b/deploy/helm/openshell/templates/clusterrole.yaml @@ -4,6 +4,7 @@ {{ $kubernetesConfig := include "openshell.effectiveKubernetesConfig" . | fromYaml -}} {{- $workspaceMode := get $kubernetesConfig "workspace_mode" | default "shared" }} +{{- $supervisorNetwork := .Values.supervisor.network | default dict }} {{- $allowDriverConfig := get $kubernetesConfig "allow_driver_config" | default false -}} {{- $managedSshIngressEnabled := .Values.networkPolicy.enabled -}} {{- $gatewayConfig := .Values.gatewayConfig | default dict -}} @@ -98,6 +99,23 @@ rules: - patch - watch {{- end }} + {{- if and (ne $workspaceMode "shared") (get $supervisorNetwork "additionalCaConfigMapName") }} + # The Kubernetes driver creates and verifies immutable, content-addressed + # destination-CA ConfigMaps. Kubernetes RBAC supports exact resourceNames + # only, not the digest-bearing name prefix, and CREATE cannot be restricted + # by resourceNames. This ClusterRole is bound cluster-wide in managed and + # operator modes, so keep the required capability to get/create only: the + # driver never lists, watches, patches, updates, or deletes these objects. + # Use a dedicated gateway service account and namespace-scoped bindings when + # this read boundary matters. + - apiGroups: + - "" + resources: + - configmaps + verbs: + - get + - create + {{- end }} {{- if ne $workspaceMode "shared" }} - apiGroups: [""] resources: ["services"] diff --git a/deploy/helm/openshell/templates/role.yaml b/deploy/helm/openshell/templates/role.yaml index 4097976ce..85d2e954d 100644 --- a/deploy/helm/openshell/templates/role.yaml +++ b/deploy/helm/openshell/templates/role.yaml @@ -1,7 +1,9 @@ {{- $kubernetesConfig := include "openshell.effectiveKubernetesConfig" . | fromYaml -}} {{- $workspaceMode := get $kubernetesConfig "workspace_mode" | default "shared" -}} {{- $allowDriverConfig := get $kubernetesConfig "allow_driver_config" | default false -}} -{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) (include "openshell.rbacCreate" .) }} +{{- $supervisorNetwork := .Values.supervisor.network | default dict -}} +{{- $additionalCaEnabled := ne (get $supervisorNetwork "additionalCaConfigMapName" | default "") "" -}} +{{- if and (eq $workspaceMode "shared") (or (include "openshell.workspaceResourcesEnabled" .) $additionalCaEnabled) (include "openshell.rbacCreate" .) }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 apiVersion: rbac.authorization.k8s.io/v1 @@ -12,6 +14,7 @@ metadata: labels: {{- include "openshell.labels" . | nindent 4 }} rules: + {{- if include "openshell.workspaceResourcesEnabled" . }} {{- if $allowDriverConfig }} # Metadata-only admission of caller-selected PVCs requires get permission. - apiGroups: [""] @@ -67,4 +70,13 @@ rules: - apiGroups: ["networking.k8s.io"] resources: ["networkpolicies"] verbs: ["create", "get"] + {{- end }} + {{- if $additionalCaEnabled }} + # Immutable, content-addressed destination-CA generations are verified by + # exact name. The driver intentionally has no list/watch/patch/update/delete + # capability for ConfigMaps, so retained generations cannot be mutated. + - apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "create"] + {{- end }} {{- end }} diff --git a/deploy/helm/openshell/templates/rolebinding.yaml b/deploy/helm/openshell/templates/rolebinding.yaml index a81d8c56c..ab487477c 100644 --- a/deploy/helm/openshell/templates/rolebinding.yaml +++ b/deploy/helm/openshell/templates/rolebinding.yaml @@ -1,6 +1,8 @@ {{- $kubernetesConfig := include "openshell.effectiveKubernetesConfig" . | fromYaml -}} {{- $workspaceMode := get $kubernetesConfig "workspace_mode" | default "shared" -}} -{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) (include "openshell.rbacCreate" .) }} +{{- $supervisorNetwork := .Values.supervisor.network | default dict -}} +{{- $additionalCaEnabled := ne (get $supervisorNetwork "additionalCaConfigMapName" | default "") "" -}} +{{- if and (eq $workspaceMode "shared") (include "openshell.rbacCreate" .) (or (include "openshell.workspaceResourcesEnabled" .) $additionalCaEnabled) }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 apiVersion: rbac.authorization.k8s.io/v1 diff --git a/deploy/helm/openshell/tests/gateway_config_test.yaml b/deploy/helm/openshell/tests/gateway_config_test.yaml index 6922fc4a0..16639d892 100644 --- a/deploy/helm/openshell/tests/gateway_config_test.yaml +++ b/deploy/helm/openshell/tests/gateway_config_test.yaml @@ -1294,6 +1294,25 @@ tests: - failedTemplate: errorPattern: "replicaCount > 1 requires server.externalDbSecret" + - it: renders writable ephemeral XDG state for gateway trust artifacts + template: templates/statefulset.yaml + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: XDG_STATE_HOME + value: /var/lib/openshell-state + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: gateway-state + mountPath: /var/lib/openshell-state + - contains: + path: spec.template.spec.volumes + content: + name: gateway-state + emptyDir: {} + - it: renders a Deployment for external database-backed gateway workloads template: templates/deployment.yaml set: @@ -1331,6 +1350,16 @@ tests: secretKeyRef: name: my-pg-secret key: uri + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: gateway-state + mountPath: /var/lib/openshell-state + - contains: + path: spec.template.spec.volumes + content: + name: gateway-state + emptyDir: {} - it: fails when a Deployment uses the default SQLite database template: templates/statefulset.yaml diff --git a/deploy/helm/openshell/tests/network_additional_ca_test.yaml b/deploy/helm/openshell/tests/network_additional_ca_test.yaml new file mode 100644 index 000000000..3b3208483 --- /dev/null +++ b/deploy/helm/openshell/tests/network_additional_ca_test.yaml @@ -0,0 +1,223 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: network additional CA delivery +release: + name: openshell + namespace: my-namespace +templates: + - templates/gateway-config.yaml + - templates/statefulset.yaml + - templates/role.yaml + - templates/rolebinding.yaml + - templates/clusterrole.yaml + +tests: + - it: omits the global additional CA setting by default + template: templates/gateway-config.yaml + asserts: + - notMatchRegex: + path: data["gateway.toml"] + pattern: '\[openshell\.supervisor\.network\]' + + - it: treats a missing supervisor network map as disabled + template: templates/gateway-config.yaml + set: + supervisor.network: null + asserts: + - notMatchRegex: + path: data["gateway.toml"] + pattern: '\[openshell\.supervisor\.network\]' + + - it: omits the gateway source mount by default + template: templates/statefulset.yaml + asserts: + - notContains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: additional-ca-source + any: true + - notContains: + path: spec.template.spec.volumes + content: + name: additional-ca-source + any: true + + - it: omits shared namespace ConfigMap permissions by default + template: templates/role.yaml + asserts: + - notContains: + path: rules + content: + apiGroups: [""] + resources: ["configmaps"] + any: true + + - it: omits cluster-wide ConfigMap permissions by default + template: templates/clusterrole.yaml + asserts: + - notContains: + path: rules + content: + apiGroups: [""] + resources: ["configmaps"] + any: true + + - it: renders the global setting outside the Kubernetes driver table + template: templates/gateway-config.yaml + set: + supervisor.network.additionalCaConfigMapName: private-destination-ca + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: '(?ms)\[openshell\.supervisor\.network\]\s*additional_ca_bundle\s*=\s*"/etc/openshell-tls/additional-ca-source/ca\.crt"' + - notMatchRegex: + path: data["gateway.toml"] + pattern: '(?ms)\[openshell\.drivers\.kubernetes\][^\[]*additional_ca_bundle' + + - it: mounts the operator source ConfigMap into the gateway + template: templates/statefulset.yaml + set: + supervisor.network.additionalCaConfigMapName: private-destination-ca + asserts: + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: additional-ca-source + mountPath: /etc/openshell-tls/additional-ca-source/ca.crt + subPath: ca.crt + readOnly: true + - contains: + path: spec.template.spec.volumes + content: + name: additional-ca-source + configMap: + name: private-destination-ca + items: + - key: ca.crt + path: ca.crt + - exists: + path: spec.template.metadata.annotations["checksum/gateway-config"] + + - it: grants only namespace-wide ConfigMap get and create in shared mode + template: templates/role.yaml + set: + supervisor.network.additionalCaConfigMapName: private-destination-ca + workspaceResources.enabled: false + asserts: + - contains: + path: rules + content: + apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "create"] + - notContains: + path: rules + content: + apiGroups: [""] + resources: ["configmaps"] + resourceNames: ["openshell-additional-ca-openshell"] + any: true + - notContains: + path: rules + content: + apiGroups: ["agents.x-k8s.io"] + any: true + + - it: binds shared namespace ConfigMap permissions when workspace resources are external + template: templates/rolebinding.yaml + set: + supervisor.network.additionalCaConfigMapName: private-destination-ca + workspaceResources.enabled: false + asserts: + - hasDocuments: + count: 1 + + - it: grants only ConfigMap get/create when shared workspace resources are external + template: templates/role.yaml + set: + supervisor.network.additionalCaConfigMapName: private-destination-ca + workspaceResources.enabled: false + asserts: + - contains: + path: rules + content: + apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "create"] + - notContains: + path: rules + content: + apiGroups: [""] + resources: ["pods"] + any: true + - notContains: + path: rules + content: + apiGroups: [""] + resources: ["secrets"] + any: true + - notContains: + path: rules + content: + apiGroups: ["networking.k8s.io"] + resources: ["networkpolicies"] + any: true + + - it: grants only cluster-wide ConfigMap get and create in managed mode + template: templates/clusterrole.yaml + set: + supervisor.network.additionalCaConfigMapName: private-destination-ca + server.drivers.kubernetes.workspaceMode: managed + asserts: + - contains: + path: rules + content: + apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "create"] + - notContains: + path: rules + content: + apiGroups: [""] + resources: ["configmaps"] + resourceNames: ["openshell-additional-ca-openshell"] + any: true + + - it: omits a shared Role in managed workspace mode + template: templates/role.yaml + set: + supervisor.network.additionalCaConfigMapName: private-destination-ca + server.drivers.kubernetes.workspaceMode: managed + asserts: + - hasDocuments: + count: 0 + + - it: grants only cluster-wide ConfigMap get and create in operator mode + template: templates/clusterrole.yaml + set: + supervisor.network.additionalCaConfigMapName: private-destination-ca + server.drivers.kubernetes.workspaceMode: operator + asserts: + - contains: + path: rules + content: + apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "create"] + - notContains: + path: rules + content: + apiGroups: [""] + resources: ["configmaps"] + resourceNames: ["openshell-additional-ca-openshell"] + any: true + + - it: omits a shared Role in operator workspace mode + template: templates/role.yaml + set: + supervisor.network.additionalCaConfigMapName: private-destination-ca + server.drivers.kubernetes.workspaceMode: operator + asserts: + - hasDocuments: + count: 0 diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index af4d44266..42eec1a39 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -134,6 +134,9 @@ supervisor: # gatewayConfig selects supervisor_topology = "gateway". Top-level affinity # applies only to the gateway pod. affinity: {} + network: + # -- Name of an operator-owned ConfigMap with a `ca.crt` key containing additional destination CA certificates for sandbox supervisors. Empty disables additional destination trust. + additionalCaConfigMapName: "" sandboxRuntime: # -- Workload boundary TLS listener port. boundaryPort: 5500 diff --git a/docs/how-it-works/gateways/configuration.mdx b/docs/how-it-works/gateways/configuration.mdx index 0876955cb..b3ba0b470 100644 --- a/docs/how-it-works/gateways/configuration.mdx +++ b/docs/how-it-works/gateways/configuration.mdx @@ -90,6 +90,9 @@ version = 2 [openshell.gateway.oidc] # ... JWT bearer auth ... +[openshell.supervisor.network] +# ... shared network-supervisor destination trust ... + [openshell.drivers.kubernetes] # ... driver-specific settings ... @@ -99,6 +102,95 @@ version = 2 The gateway selector is `compute_driver = ""`. It accepts one scalar driver name. Omit it to retain auto-detection. +## Additional Destination CAs + +Use the global supervisor network table to trust private certificate authorities +for sandbox egress destinations: + +```toml +[openshell.supervisor.network] +additional_ca_bundle = "/etc/openshell/certs/private-destination-ca.pem" +``` + +The path refers to one pre-assembled PEM bundle on the gateway host or in the +gateway container. It must contain one or more PEM certificate blocks. Inline +PEM, private keys, DER-only input, URLs, empty files, unrelated text, mixed +certificate/key files, symlinks, and non-regular files are unsupported. The +gateway reads the bundle once at startup, validates and canonicalizes it into a +gateway-owned artifact, and fails startup with the field and source path when +material is missing or unusable. It does not include certificate contents in +diagnostics. + +The normalized certificates augment the default bundled, system, or native +roots used for sandbox destination TLS. Normal certificate-chain and hostname +verification remains enabled. A configured CA can authenticate every +policy-permitted endpoint whose certificate chains to that CA and matches the +requested hostname, so scope private CAs accordingly. + +This setting is separate from all other trust domains: + +- `proxy_ca_bundle` authenticates a configured corporate proxy and its TLS + interception path. It still requires the corresponding proxy setting. +- `guest_tls_*`, `OPENSHELL_TLS_CA`, and Kubernetes client TLS Secrets + authenticate sandbox callbacks to the gateway. +- `[openshell.gateway.tls]` configures the gateway listener. +- OIDC, middleware, and interceptor CA settings authenticate their own + gateway-side connections. + +Docker and Podman bind-mount the gateway-owned artifact, Kubernetes creates an +immutable content-addressed managed ConfigMap, and VM stages its snapshot in the +sandbox overlay. Each mounts it at +`/etc/openshell/tls/additional/ca-bundle.pem` and starts the supervisor with +`--additional-ca-bundle` for that path. Drivers verify their staged artifact +before handing it to a runtime; the SHA-256 generation is an internal staging +value, not a supervisor argument. The supervisor +reads the mounted bundle and folds it into its system roots. Remote and custom +compute drivers reject this setting because the compute-driver protocol does +not carry the normalized material. + +### Lifecycle and certificate rollover + +This is **global gateway configuration**, not a per-sandbox setting. Source +files (or the Helm source ConfigMap) are read and normalized **only when the +gateway starts**. Updating a source file or ConfigMap alone has no effect: +OpenShell does not watch or hot-reload those sources, and it does not +automatically restart sandbox consumers. A running supervisor retains the +startup trust bundle it already loaded. + +To apply any addition, removal, or source update, the operator must: + +1. Update the source files or source ConfigMap. +2. Restart or redeploy the gateway so it reads and normalizes the new source. +3. Explicitly stop and then start **every affected sandbox**. + +Stop/start retains the sandbox identity and its supported durable workspace; it +does not create a replacement sandbox. The restarted supervisor consumes the +bundle staged from the restarted gateway's snapshot. Source normalization, +driver delivery, and supervisor validation failures fail closed: +they do not silently fall back to a previous or default trust bundle. + +Use this sequence for a non-disruptive private-CA rollover: + +1. Configure both the old and new roots. +2. Restart or redeploy the gateway, then stop/start every affected sandbox. +3. Rotate destination certificates to chain to the new root. +4. Remove the old root, then repeat the gateway restart/redeploy and + stop/start sequence for every affected sandbox. + +For Helm, create an operator-owned ConfigMap with a `ca.crt` key and set +`supervisor.network.additionalCaConfigMapName`. The chart mounts the source at +`/etc/openshell-tls/additional-ca-source/ca.crt`, renders that path in +the global table, and grants the Kubernetes driver ConfigMap `get` and `create` +permissions for the selected workspace mode. These permissions cover the shared +sandbox namespace in shared mode and are cluster-wide through the existing +ClusterRoleBinding in managed and operator modes. Kubernetes RBAC cannot limit +`create` by resource name or limit `get` to the digest-bearing name prefix. The +chart does not grant ConfigMap `list`, `watch`, `patch`, `update`, or `delete` +for this feature. Use a dedicated gateway service account and namespace-scoped +bindings when this read boundary matters. Updating only the source ConfigMap +does not trigger the gateway workload checksum, so explicitly restart or +redeploy the gateway and then stop/start every affected sandbox. + ## Migrate to schema version 2 Schema version 2 is an intentional breaking cutover. The gateway rejects files @@ -229,6 +321,11 @@ provider_profile_sources = [ { type = "user" }, ] +# Additional private destination roots are global supervisor settings, not +# compute-driver settings. The gateway reads certificate-only PEM at startup. +[openshell.supervisor.network] +additional_ca_bundle = "/etc/openshell/certs/private-destination-ca.pem" + # Operator-run supervisor middleware. The gRPC endpoint must be reachable from # both the gateway and sandbox supervisors. [[openshell.supervisor.middleware]] @@ -1379,22 +1476,6 @@ health_check_interval_secs = 10 # proxy_connect_by_hostname = true # Corporate CA trusted for an https:// proxy and TLS-intercepting proxies. # proxy_ca_bundle = "/etc/openshell/tls/proxy-ca.pem" -# additional_ca_bundle points at a PEM CA bundle (on the gateway host) trusted -# for direct, policy-inspected HTTPS egress to a destination signed by a -# private CA — independent of proxy_ca_bundle above, this applies whether or -# not https_proxy is set. The Podman supervisor runs as its own container, -# separate from the workload image, so it cannot inherit a private CA baked -# into the workload's own trust store; this setting restores that trust -# without rebuilding the supervisor image. The gateway bind-mounts the file -# read-only into the supervisor container and folds it into the upstream -# trust store alongside (not instead of) the system CA bundle. The CA is also -# added to the trust bundle inside each sandbox, so workloads trust it directly. -# It is trusted for every destination, not only the one that needs it, so use a -# CA you control or one with name constraints. The file must -# exist and contain at least one certificate; the gateway validates this on -# the gateway host when the Podman driver starts, so a bad path or a -# certificate-free bundle fails gateway startup rather than a later sandbox. -# additional_ca_bundle = "/etc/openshell/tls/additional-ca.pem" # Project a host Workload API Unix socket into the supervisor, or use an # explicit container-reachable TCP endpoint, for provider token exchange. # provider_spiffe_workload_api_socket = "/run/spire/agent.sock" diff --git a/e2e/rust/Cargo.toml b/e2e/rust/Cargo.toml index d903c2e6e..4476485f7 100644 --- a/e2e/rust/Cargo.toml +++ b/e2e/rust/Cargo.toml @@ -26,6 +26,7 @@ e2e = [] # when it has resolved an IP. e2e-host-gateway = ["e2e"] e2e-local-container-driver = ["e2e"] +e2e-additional-ca = ["e2e", "e2e-host-gateway"] e2e-docker = ["e2e", "e2e-host-gateway", "e2e-local-container-driver"] # Selects tests that need a Docker gateway with config_delivery_mode = "push". e2e-config-push = ["e2e-docker"] @@ -98,6 +99,11 @@ name = "docker_corporate_proxy" path = "tests/docker_corporate_proxy.rs" required-features = ["e2e-docker"] +[[test]] +name = "additional_ca" +path = "tests/additional_ca.rs" +required-features = ["e2e-additional-ca"] + [[test]] name = "driver_config_volume" path = "tests/driver_config_volume.rs" diff --git a/e2e/rust/e2e-docker.sh b/e2e/rust/e2e-docker.sh index 5a502dd8e..8db220ca1 100755 --- a/e2e/rust/e2e-docker.sh +++ b/e2e/rust/e2e-docker.sh @@ -18,6 +18,11 @@ RUN_WITH_GATEWAY_COMMAND="__openshell_run_docker_e2e" # shellcheck source=e2e/support/conformance.sh source "${ROOT}/e2e/support/conformance.sh" +if [ "${E2E_TEST}" = "additional_ca" ]; then + export OPENSHELL_E2E_ADDITIONAL_CA=1 + E2E_FEATURES="${E2E_FEATURES},e2e-additional-ca" +fi + run_e2e() { e2e_run_openshell_conformance "Docker" diff --git a/e2e/rust/e2e-kubernetes.sh b/e2e/rust/e2e-kubernetes.sh index e17244d9e..1ffa79cea 100755 --- a/e2e/rust/e2e-kubernetes.sh +++ b/e2e/rust/e2e-kubernetes.sh @@ -27,8 +27,16 @@ RUN_WITH_GATEWAY_COMMAND="__openshell_run_kubernetes_e2e" source "${ROOT}/e2e/support/conformance.sh" # Unset or empty selects Rust defaults; e2e-conformance-only skips Rust tests. +# Reusable-workflow inputs are exported as an explicitly empty environment +# variable when the caller uses their default. Treat that the same as unset so +# the focused additional-CA row retains the upstream Kubernetes feature set. E2E_FEATURES="${OPENSHELL_E2E_KUBERNETES_FEATURES:-e2e,e2e-host-gateway,e2e-kubernetes}" +if [ "${OPENSHELL_E2E_KUBE_TEST:-}" = "additional_ca" ]; then + export OPENSHELL_E2E_ADDITIONAL_CA=1 + E2E_FEATURES="${E2E_FEATURES},e2e-additional-ca" +fi + # Fixed output path of the `e2e-kubernetes` nextest profile (`.config/nextest.toml`). JUNIT_XML="${ROOT}/results/e2e-kubernetes.xml" diff --git a/e2e/rust/e2e-podman.sh b/e2e/rust/e2e-podman.sh index 6502fb7de..06cbccc8f 100755 --- a/e2e/rust/e2e-podman.sh +++ b/e2e/rust/e2e-podman.sh @@ -67,6 +67,11 @@ if [ -n "${E2E_TEST}" ]; then SELECTED_TESTS=("${E2E_TEST}") fi +if [ "${E2E_TEST}" = "additional_ca" ]; then + export OPENSHELL_E2E_ADDITIONAL_CA=1 + E2E_FEATURES="${E2E_FEATURES},e2e-additional-ca" +fi + if [ "${E2E_TEST}" = "gpu" ] && [ -z "${OPENSHELL_E2E_WORKLOAD_MANIFEST:-}" ] && [ ! -f "${DEFAULT_WORKLOAD_MANIFEST}" ]; then echo "note: running Podman GPU e2e without a workload manifest; workload validation will log an explicit skip. Build one with 'CONTAINER_ENGINE=podman mise run e2e:workloads:build' or set OPENSHELL_E2E_WORKLOAD_MANIFEST." fi diff --git a/e2e/rust/e2e-vm.sh b/e2e/rust/e2e-vm.sh index 59a219210..0cb2c4685 100755 --- a/e2e/rust/e2e-vm.sh +++ b/e2e/rust/e2e-vm.sh @@ -52,6 +52,11 @@ CLI_BIN="${OPENSHELL_BIN:-${ROOT}/target/debug/openshell}" E2E_TEST_OVERRIDE="${OPENSHELL_E2E_VM_TEST:-}" # Unset or empty selects Rust defaults; e2e-conformance-only skips Rust tests. E2E_FEATURES="${OPENSHELL_E2E_VM_FEATURES:-e2e-vm}" +ADDITIONAL_CA_MODE="${OPENSHELL_E2E_ADDITIONAL_CA:-0}" +if [ "${E2E_TEST_OVERRIDE}" = "additional_ca" ]; then + ADDITIONAL_CA_MODE=1 + E2E_FEATURES="${OPENSHELL_E2E_VM_FEATURES:-e2e-vm,e2e-additional-ca}" +fi SANDBOX_IMAGE="${OPENSHELL_SANDBOX_IMAGE:-nvcr.io/nvidia/base/ubuntu:24.04}" # The VM driver places `compute-driver.sock` under `[openshell.drivers.vm].state_dir`. @@ -169,8 +174,13 @@ s.close()')" # basename short — see the SUN_LEN comment above. RUN_STATE_DIR="${STATE_DIR_ROOT}/os-vm-e2e-${HOST_PORT}-$$" mkdir -p "${RUN_STATE_DIR}" +# Keep every gateway-created artifact, including normalized additional CA +# material, under this invocation's disposable state root. Set this before +# starting either fixture or gateway so no host-global XDG fallback is used. export XDG_CONFIG_HOME="${RUN_STATE_DIR}/config" export XDG_DATA_HOME="${RUN_STATE_DIR}/data" +export XDG_STATE_HOME="${RUN_STATE_DIR}/state" +mkdir -p "${XDG_CONFIG_HOME}" "${XDG_DATA_HOME}" "${XDG_STATE_HOME}" GATEWAY_LOG="$(mktemp /tmp/openshell-gateway-e2e.XXXXXX)" GATEWAY_PID_FILE="${RUN_STATE_DIR}/gateway.pid" @@ -183,6 +193,10 @@ GATEWAY_NAME="openshell-e2e-vm-${HOST_PORT}" DRIVER_PID="" DRIVER_LOG="${RUN_STATE_DIR}/vm-driver.log" DRIVER_SOCKET="${RUN_STATE_DIR}/compute-driver.sock" +ADDITIONAL_CA_SERVER_PID="" +ADDITIONAL_CA_SERVER_LOG="${RUN_STATE_DIR}/additional-ca-server.log" +ADDITIONAL_CA_DIR="${RUN_STATE_DIR}/additional-ca" +ADDITIONAL_CA_PORT="" # ── Cleanup (trap) ─────────────────────────────────────────────────── @@ -207,6 +221,7 @@ cleanup() { wait "${gateway_pid}" 2>/dev/null || true fi e2e_stop_process "${DRIVER_PID}" "external VM compute driver" + e2e_stop_process "${ADDITIONAL_CA_SERVER_PID}" "additional CA HTTPS fixture" # On failure, keep the VM console log for debugging. We deliberately # print it instead of leaving it on disk because the state dir gets @@ -242,6 +257,19 @@ trap cleanup EXIT # ── Launch the gateway + VM driver ─────────────────────────────────── +start_additional_ca_fixture() { + e2e_start_additional_ca_fixture \ + "${ADDITIONAL_CA_DIR}" "${ADDITIONAL_CA_SERVER_LOG}" \ + ADDITIONAL_CA_SERVER_PID ADDITIONAL_CA_PORT +} +if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then + if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then + echo "ERROR: additional CA e2e requires the gateway-managed VM driver." >&2 + exit 2 + fi + start_additional_ca_fixture +fi + echo "==> Starting openshell-gateway on 127.0.0.1:${HOST_PORT} (state: ${RUN_STATE_DIR})" # Pin `driver_dir` to the workspace `target/debug/` so we always pick up @@ -280,6 +308,15 @@ gateway_id = "${GATEWAY_NAME}" # Local VM e2e gateways exercise the single-player default: sandbox JWTs # identify the supervisor and do not expire. +EOF +if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then + cat >>"${GATEWAY_CONFIG}" <>"${GATEWAY_CONFIG}" <&2 + exit 1 + fi + # Both paths exist after gateway initialization; plain `realpath` is + # portable across GNU and BSD/macOS implementations. + state_root="$(realpath "${XDG_STATE_HOME}")" + artifact_path="$(realpath "${additional_ca_artifacts[0]}")" + case "${artifact_path}" in + "${state_root}"/*) ;; + *) + echo "ERROR: normalized additional CA artifact escaped XDG_STATE_HOME" >&2 + exit 1 + ;; + esac + export OPENSHELL_E2E_ADDITIONAL_CA_ARTIFACT="${artifact_path}" + export OPENSHELL_E2E_GATEWAY_CONFIG="${GATEWAY_CONFIG}" +fi e2e_export_gateway_restart_metadata \ "${GATEWAY_BIN}" \ "${GATEWAY_ARGS_FILE}" \ diff --git a/e2e/rust/tests/additional_ca.rs b/e2e/rust/tests/additional_ca.rs new file mode 100644 index 000000000..99599b37d --- /dev/null +++ b/e2e/rust/tests/additional_ca.rs @@ -0,0 +1,1232 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Driver-backed e2e coverage for the global network-supervisor additional CA. + +use std::io::Write; +use std::net::TcpListener; +use std::process::{Child, Command, Stdio}; +use std::time::Duration; + +use openshell_e2e::harness::cli::{run_cli, wait_for_healthy, wait_for_sandbox_phase}; +use openshell_e2e::harness::gateway::ManagedGateway; +use openshell_e2e::harness::sandbox::SandboxGuard; +use serde_json::Value; +use sha2::Digest as _; +use tempfile::NamedTempFile; + +#[cfg(any(feature = "e2e-docker", feature = "e2e-podman"))] +use openshell_e2e::harness::container::ContainerEngine; + +const ADDITIONAL_CA_MOUNT_PATH: &str = "/etc/openshell/tls/additional/ca-bundle.pem"; +#[cfg(any(feature = "e2e-docker", feature = "e2e-podman"))] +const GATEWAY_CA_PATH: &str = "/etc/openshell/tls/client/ca.crt"; +const READY_MARKER: &str = "additional-ca-e2e-ready"; +const LIFECYCLE_READY_MARKER: &str = "additional-ca-lifecycle-ready"; +const LIFECYCLE_DURABLE_MARKER: &str = "additional-ca-lifecycle-durable"; +const LIFECYCLE_DURABLE_PATH: &str = "/sandbox/additional-ca-lifecycle-marker"; +const ADDITIONAL_CA_VOLUME: &str = "openshell-additional-ca"; + +struct PortForwardGuard { + child: Child, +} + +impl Drop for PortForwardGuard { + fn drop(&mut self) { + let _ = self.child.kill(); + let _ = self.child.wait(); + } +} + +fn driver() -> Result<&'static str, String> { + match std::env::var("OPENSHELL_E2E_DRIVER").as_deref() { + Ok("docker") => Ok("docker"), + Ok("podman") => Ok("podman"), + Ok("kubernetes") => Ok("kubernetes"), + Ok("vm") => Ok("vm"), + Ok(other) => Err(format!("unsupported additional CA e2e driver: {other}")), + Err(error) => Err(format!("OPENSHELL_E2E_DRIVER must be set: {error}")), + } +} + +const MISMATCH_HOST: &str = "mismatch.openshell.internal"; + +fn write_policy(port: u16) -> Result { + let mut file = NamedTempFile::new().map_err(|error| format!("create policy: {error}"))?; + let policy = format!( + r#"version: 1 + +filesystem_policy: + include_workdir: true + read_only: [/usr, /lib, /proc, /dev/urandom, /app, /etc, /var/log] + read_write: [/sandbox, /tmp, /dev/null] + +landlock: + compatibility: best_effort + +process: + run_as_user: sandbox + run_as_group: sandbox + +network_policies: + additional_ca_e2e: + name: additional_ca_e2e + endpoints: + - host: host.openshell.internal + port: {port} + path: /** + protocol: rest + access: full + enforcement: enforce + allowed_ips: ["10.0.0.0/8", "172.0.0.0/8", "192.168.0.0/16", "fc00::/7"] + - host: {MISMATCH_HOST} + port: {port} + path: /** + protocol: rest + access: full + enforcement: enforce + allowed_ips: ["10.0.0.0/8", "172.0.0.0/8", "192.168.0.0/16", "fc00::/7"] + - host: example.com + port: 443 + path: /** + protocol: rest + access: full + enforcement: enforce + binaries: + # The network-policy binary identity is the resolved /proc//exe + # target, so the versioned real binaries must be allowlisted rather + # than the unversioned symlinks: python3.12 on the Noble fixture image + # (/usr/bin/python3 -> python3.12) and on the uv-based Kubernetes/VM + # workload image (/usr/local/bin/python3 -> python3.12). + - path: /usr/bin/python3.12 + - path: /usr/local/bin/python3.12 +"# + ); + file.write_all(policy.as_bytes()) + .map_err(|error| format!("write policy: {error}"))?; + file.flush() + .map_err(|error| format!("flush policy: {error}"))?; + Ok(file) +} + +fn run_json(mut command: Command, description: &str) -> Result { + let output = command + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .output() + .map_err(|error| format!("run {description}: {error}"))?; + let stdout = String::from_utf8_lossy(&output.stdout); + let stderr = String::from_utf8_lossy(&output.stderr); + if !output.status.success() { + return Err(format!("{description} failed:\n{stdout}{stderr}")); + } + serde_json::from_str(stdout.trim()) + .map_err(|error| format!("parse {description} JSON: {error}; output={stdout}")) +} + +#[cfg(any(feature = "e2e-docker", feature = "e2e-podman"))] +fn engine_command(container_engine: &ContainerEngine) -> Command { + let mut command = container_engine.command(); + if container_engine.name() == "podman" + && let Ok(socket) = std::env::var("OPENSHELL_PODMAN_SOCKET") + { + command.arg("--url").arg(format!("unix://{socket}")); + } + command +} + +#[cfg(any(feature = "e2e-docker", feature = "e2e-podman"))] +fn inspect_local_container(driver: &str, sandbox_name: &str) -> Result<(), String> { + let container_engine = ContainerEngine::from_env().map_err(|error| error.clone())?; + // The supervisor container owns the additional CA mount and command. It is + // host-networked (or attached to a runtime bridge), so it cannot be + // located through e2e network membership; select it by managed labels. + // The sandbox namespace label is not used because the Podman driver + // labels containers with the gateway-elided request namespace. + let output = engine_command(&container_engine) + .args([ + "ps", + "-aq", + "--filter", + "label=openshell.ai/isolation-role=supervisor", + "--filter", + ]) + .arg(format!("label=openshell.ai/sandbox-name={sandbox_name}")) + .output() + .map_err(|error| format!("run {driver} ps: {error}"))?; + if !output.status.success() { + return Err(format!( + "{driver} ps failed: {}", + String::from_utf8_lossy(&output.stderr) + )); + } + let ids = String::from_utf8_lossy(&output.stdout) + .lines() + .map(str::trim) + .filter(|line| !line.is_empty()) + .map(ToString::to_string) + .collect::>(); + let [container_id] = ids.as_slice() else { + return Err(format!( + "expected one {driver} supervisor container for {sandbox_name}, found {ids:?}" + )); + }; + + let mut mounts_command = engine_command(&container_engine); + mounts_command.args(["inspect", "--format", "{{json .Mounts}}", container_id]); + let mounts = run_json(mounts_command, &format!("{driver} mount inspection"))?; + let mounts = mounts.as_array().ok_or("mounts must be an array")?; + let destination_mounts = mounts + .iter() + .filter(|mount| mount["Destination"] == ADDITIONAL_CA_MOUNT_PATH) + .collect::>(); + if destination_mounts.len() != 1 || destination_mounts[0]["RW"] != false { + return Err(format!("unexpected destination CA mounts: {mounts:#?}")); + } + // Podman bind-mounts the gateway mTLS CA only when user namespaces are + // not remapping UIDs; otherwise it is delivered as a container secret + // (`openshell-tls-ca-*`), which does not appear in `.Mounts`. Docker never + // bind-mounts it: the gateway CA is staged inside the read-only supervisor + // state volume as `tls/ca.pem` and the supervisor is pointed at it through + // the `OPENSHELL_TLS_CA` environment variable. + let gateway_ca_mount = mounts + .iter() + .find(|mount| mount["Destination"] == GATEWAY_CA_PATH); + let gateway_ca_secret = gateway_ca_mount.is_none() + && driver == "podman" + && podman_gateway_ca_secret(&container_engine, driver, container_id)?; + let gateway_ca_staged = gateway_ca_mount.is_none() + && driver == "docker" + && docker_gateway_ca_staged(&container_engine, container_id, mounts)?; + if gateway_ca_mount.is_none() && !gateway_ca_secret && !gateway_ca_staged { + return Err("gateway mTLS CA delivery must remain present".to_string()); + } + let gateway_ca_source = gateway_ca_mount.map(|mount| mount["Source"].clone()); + if let Some(gateway_ca_source) = gateway_ca_source + && destination_mounts[0]["Source"] == gateway_ca_source + { + return Err("destination and gateway CA mounts must be distinct".to_string()); + } + + let mut command_inspect = engine_command(&container_engine); + command_inspect.args(["inspect", "--format", "{{json .Config.Cmd}}", container_id]); + let command = run_json(command_inspect, &format!("{driver} command inspection"))?; + assert_network_ca_command(command.as_array().ok_or("command must be an array")?) +} + +/// Check whether the supervisor received the gateway mTLS CA as a Podman +/// container secret (the delivery path used when user namespaces remap UIDs). +#[cfg(any(feature = "e2e-docker", feature = "e2e-podman"))] +fn podman_gateway_ca_secret( + container_engine: &ContainerEngine, + driver: &str, + container_id: &str, +) -> Result { + let mut secrets_command = engine_command(container_engine); + secrets_command.args([ + "inspect", + "--format", + "{{json .Config.Secrets}}", + container_id, + ]); + let secrets = run_json(secrets_command, &format!("{driver} secret inspection"))?; + let secrets = secrets + .as_array() + .ok_or("supervisor secrets must be an array")?; + Ok(secrets.iter().any(|secret| { + secret["Name"] + .as_str() + .is_some_and(|name| name.starts_with("openshell-tls-ca-")) + })) +} + +/// Check whether the Docker supervisor received the gateway mTLS CA through +/// its staged delivery path: the read-only supervisor state volume plus the +/// `OPENSHELL_TLS_CA` environment pointer. +#[cfg(any(feature = "e2e-docker", feature = "e2e-podman"))] +fn docker_gateway_ca_staged( + container_engine: &ContainerEngine, + container_id: &str, + mounts: &[Value], +) -> Result { + const SUPERVISOR_STATE_MOUNT_PATH: &str = "/.openshell/supervisor"; + let supervisor_volume = mounts + .iter() + .find(|mount| mount["Destination"] == SUPERVISOR_STATE_MOUNT_PATH); + if supervisor_volume.is_none_or(|mount| mount["RW"] != false) { + return Ok(false); + } + let mut env_command = engine_command(container_engine); + env_command.args(["inspect", "--format", "{{json .Config.Env}}", container_id]); + let envs = run_json(env_command, "docker environment inspection")?; + let envs = envs + .as_array() + .ok_or("supervisor environment must be an array")?; + Ok(envs + .iter() + .any(|env| env.as_str() == Some("OPENSHELL_TLS_CA=/.openshell/supervisor/tls/ca.pem"))) +} + +fn assert_additional_ca_command(command: &[Value]) -> Result<(), String> { + let bundle_pairs = command + .windows(2) + .filter(|pair| pair[0] == "--additional-ca-bundle") + .collect::>(); + if bundle_pairs.len() != 1 || bundle_pairs[0][1] != ADDITIONAL_CA_MOUNT_PATH { + return Err(format!( + "unexpected additional CA command arguments: {command:#?}" + )); + } + Ok(()) +} + +#[cfg(any(feature = "e2e-docker", feature = "e2e-podman"))] +fn assert_network_ca_command(command: &[Value]) -> Result<(), String> { + assert_additional_ca_command(command) +} + +fn kubectl_json(args: &[&str]) -> Result { + let context = std::env::var("OPENSHELL_E2E_KUBE_CONTEXT_ACTIVE") + .map_err(|error| format!("active Kubernetes context missing: {error}"))?; + let mut command = Command::new("kubectl"); + command.arg("--context").arg(context).args(args); + run_json(command, "kubectl") +} + +fn inspect_kubernetes_pod() -> Result { + let namespace = std::env::var("OPENSHELL_E2E_SANDBOX_NAMESPACE") + .map_err(|error| format!("sandbox namespace missing: {error}"))?; + let pods = kubectl_json(&["-n", &namespace, "get", "pods", "-o", "json"])?; + let items = pods["items"] + .as_array() + .ok_or("Kubernetes Pod list missing items")?; + let supervisor = items + .iter() + .find(|pod| { + pod["metadata"]["labels"]["openshell.ai/boundary-role"] == "supervisor" + && pod["spec"]["volumes"].as_array().is_some_and(|volumes| { + volumes + .iter() + .any(|volume| volume["name"] == ADDITIONAL_CA_VOLUME) + }) + }) + .ok_or("dedicated supervisor Pod with destination CA volume not found")?; + let supervisor_name = supervisor["metadata"]["name"] + .as_str() + .ok_or("dedicated supervisor Pod name missing")?; + let containers = supervisor["spec"]["containers"] + .as_array() + .ok_or("dedicated supervisor Pod containers missing")?; + let [supervisor_container] = containers.as_slice() else { + return Err(format!( + "destination CA must be mounted only in the one-container dedicated supervisor Pod: {containers:#?}" + )); + }; + if supervisor_container["name"] != "supervisor" { + return Err(format!( + "destination CA mounted in non-supervisor control container: {supervisor_container:#?}" + )); + } + let mounts = supervisor_container["volumeMounts"] + .as_array() + .ok_or("dedicated supervisor Pod volume mounts missing")?; + let destination_mounts = mounts + .iter() + .filter(|mount| mount["name"] == ADDITIONAL_CA_VOLUME) + .collect::>(); + let [destination_mount] = destination_mounts.as_slice() else { + return Err(format!( + "expected exactly one destination CA mount in dedicated supervisor Pod: {mounts:#?}" + )); + }; + if destination_mount["mountPath"] != ADDITIONAL_CA_MOUNT_PATH + || destination_mount["subPath"] != "ca.crt" + || destination_mount["readOnly"] != true + { + return Err(format!( + "dedicated supervisor destination CA mount must be a read-only ca.crt file: {destination_mount:#?}" + )); + } + assert_additional_ca_command( + supervisor_container["command"] + .as_array() + .ok_or("dedicated supervisor command missing")?, + )?; + let volume = supervisor["spec"]["volumes"] + .as_array() + .ok_or("dedicated supervisor Pod volumes missing")? + .iter() + .find(|volume| volume["name"] == ADDITIONAL_CA_VOLUME) + .ok_or("dedicated supervisor destination CA volume missing")?; + let managed_name = volume["configMap"]["name"] + .as_str() + .ok_or("managed destination CA ConfigMap name missing")?; + if !managed_name.starts_with("openshell-additional-ca-") + || volume["configMap"]["items"].as_array().is_none_or(|items| { + items != &vec![serde_json::json!({"key": "ca.crt", "path": "ca.crt"})] + }) + { + return Err(format!( + "dedicated supervisor has an unexpected destination CA ConfigMap volume: {volume:#?}" + )); + } + + let workload = items + .iter() + .find(|pod| pod["metadata"]["labels"]["openshell.ai/boundary-role"] == "workload") + .ok_or("paired workload Pod not found")?; + let workload_volumes = workload["spec"]["volumes"] + .as_array() + .ok_or("workload Pod volumes missing")?; + if workload_volumes + .iter() + .any(|volume| volume["name"] == ADDITIONAL_CA_VOLUME) + { + return Err("destination CA ConfigMap leaked into workload Pod volumes".to_string()); + } + for container in workload["spec"]["containers"] + .as_array() + .into_iter() + .flatten() + .chain( + workload["spec"]["initContainers"] + .as_array() + .into_iter() + .flatten(), + ) + { + if container["volumeMounts"].as_array().is_some_and(|mounts| { + mounts.iter().any(|mount| { + mount["name"] == ADDITIONAL_CA_VOLUME + || mount["mountPath"] == ADDITIONAL_CA_MOUNT_PATH + }) + }) { + return Err(format!( + "destination CA leaked into workload container or bootstrap init container: {container:#?}" + )); + } + } + + let config_map = kubectl_json(&[ + "-n", + &namespace, + "get", + "configmap", + managed_name, + "-o", + "json", + ])?; + let supervisor_gateway = supervisor["metadata"]["labels"]["openshell.ai/gateway-id"] + .as_str() + .ok_or("dedicated supervisor gateway ownership label missing")?; + if config_map["metadata"]["labels"]["openshell.ai/managed-by"] != "openshell" + || config_map["metadata"]["labels"]["openshell.ai/gateway-id"] != supervisor_gateway + || config_map["metadata"]["ownerReferences"].is_array() + || config_map["immutable"] != true + || config_map["data"] != serde_json::json!({"ca.crt": config_map["data"]["ca.crt"].clone()}) + || config_map["data"]["ca.crt"].as_str().is_none() + || !config_map["binaryData"].is_null() + { + return Err("managed destination CA ConfigMap ownership/data is incomplete".to_string()); + } + let config_map_digest = + config_map["metadata"]["annotations"]["openshell.ai/additional-ca-generation"] + .as_str() + .ok_or("managed destination CA ConfigMap generation missing")?; + let expected_digest = format!( + "sha256:{:x}", + sha2::Sha256::digest( + config_map["data"]["ca.crt"] + .as_str() + .unwrap_or_default() + .as_bytes(), + ) + ); + if config_map_digest != expected_digest { + return Err("managed ConfigMap generation does not match its CA data".to_string()); + } + Ok(format!("{namespace}/{managed_name}/{supervisor_name}")) +} + +#[cfg(feature = "e2e-vm")] +fn inspect_vm_overlay() -> Result<(), String> { + use sha2::{Digest, Sha256}; + use std::path::PathBuf; + + let state_dir = PathBuf::from( + std::env::var("OPENSHELL_E2E_VM_STATE_DIR") + .map_err(|error| format!("VM state directory missing: {error}"))?, + ); + let source_ca = PathBuf::from( + std::env::var("OPENSHELL_E2E_ADDITIONAL_CA_CERT") + .map_err(|error| format!("additional CA source path missing: {error}"))?, + ); + let mut sandboxes = std::fs::read_dir(state_dir.join("sandboxes")) + .map_err(|error| format!("read VM sandbox state: {error}"))? + .filter_map(Result::ok) + .map(|entry| entry.path()) + .filter(|path| path.join("overlay.ext4").is_file()) + .collect::>(); + sandboxes.sort(); + let [sandbox_state] = sandboxes.as_slice() else { + return Err(format!( + "expected one VM sandbox overlay, found {sandboxes:?}" + )); + }; + let overlay = sandbox_state.join("overlay.ext4"); + + // Host-only delivery: the driver stages the normalized destination CA as + // a driver-owned artifact beside the sandbox state, where the host + // supervisor consumes it. The guest never receives CA material through + // its overlay, rootfs, argv, or console. + let staged = std::fs::read(sandbox_state.join("additional-ca.crt")) + .map_err(|error| format!("read host-staged VM destination CA: {error}"))?; + let source = std::fs::read(&source_ca) + .map_err(|error| format!("read additional CA source for digest comparison: {error}"))?; + if Sha256::digest(&staged) != Sha256::digest(&source) { + return Err( + "host-staged VM destination CA digest does not match normalized source".to_string(), + ); + } + + // debugfs exits successfully even for missing inodes, so presence is + // detected from the "Inode:" stat header rather than the exit status. + let overlay_has = |guest_path: &str| -> Result { + let output = Command::new("debugfs") + .args(["-R", &format!("stat {guest_path}")]) + .arg(&overlay) + .output() + .map_err(|error| format!("run debugfs overlay inspection: {error}"))?; + if !output.status.success() { + return Err(format!( + "debugfs overlay inspection failed for {guest_path}: {}", + String::from_utf8_lossy(&output.stderr) + )); + } + let stdout = String::from_utf8_lossy(&output.stdout); + Ok(stdout.contains("Inode:")) + }; + for guest_path in [ + "/upper/etc/openshell-tls/additional-ca.crt", + "/etc/openshell-tls/additional-ca.crt", + ] { + if overlay_has(guest_path)? { + return Err(format!( + "VM guest overlay received CA material at {guest_path}; the \ + destination CA must stay host-only" + )); + } + } + + let console = sandbox_state.join("rootfs-console.log"); + let console = std::fs::read_to_string(&console) + .map_err(|error| format!("read VM serial log: {error}"))?; + if console.contains("-----BEGIN CERTIFICATE-----") { + return Err("VM serial log disclosed certificate material".to_string()); + } + Ok(()) +} + +fn local_staged_artifact() -> Result { + let configured = std::path::PathBuf::from( + std::env::var("OPENSHELL_E2E_ADDITIONAL_CA_ARTIFACT") + .map_err(|error| format!("gateway-owned CA artifact location missing: {error}"))?, + ); + if configured.is_file() { + return Ok(configured); + } + if !configured.is_dir() { + return Err(format!( + "gateway-owned CA artifact location does not exist: {}", + configured.display() + )); + } + + let mut artifacts = std::fs::read_dir(&configured) + .map_err(|error| { + format!( + "read gateway-owned CA artifact directory {}: {error}", + configured.display() + ) + })? + .filter_map(Result::ok) + .map(|entry| entry.path()) + .filter(|path| { + path.is_file() + && path.file_name().is_some_and(|name| { + let name = name.to_string_lossy(); + name.starts_with("additional-ca-") && name.ends_with(".crt") + }) + }) + .collect::>(); + artifacts.sort(); + let [artifact] = artifacts.as_slice() else { + return Err(format!( + "expected exactly one content-addressed CA artifact in {}, found {artifacts:?}", + configured.display() + )); + }; + Ok(artifact.clone()) +} + +#[cfg(unix)] +fn set_staged_artifact_mode(path: &std::path::Path, mode: u32) -> Result<(), String> { + use std::os::unix::fs::PermissionsExt; + + std::fs::set_permissions(path, std::fs::Permissions::from_mode(mode)) + .map_err(|error| format!("set gateway-owned CA artifact permissions: {error}")) +} + +#[cfg(not(unix))] +fn set_staged_artifact_mode(path: &std::path::Path, mode: u32) -> Result<(), String> { + let mut permissions = std::fs::metadata(path) + .map_err(|error| format!("read gateway-owned CA artifact permissions: {error}"))? + .permissions(); + permissions.set_readonly(mode & 0o222 == 0); + std::fs::set_permissions(path, permissions) + .map_err(|error| format!("set gateway-owned CA artifact permissions: {error}")) +} + +fn replace_staged_material(driver: &str, kubernetes_identity: Option<&str>) -> Result<(), String> { + if driver == "kubernetes" { + return replace_kubernetes_config_map_with_invalid_material( + kubernetes_identity.ok_or("Kubernetes staging identity missing")?, + ); + } + + let artifact = local_staged_artifact()?; + set_staged_artifact_mode(&artifact, 0o644)?; + std::fs::write(&artifact, b"not-a-certificate\n") + .map_err(|error| format!("replace gateway-owned CA artifact: {error}"))?; + set_staged_artifact_mode(&artifact, 0o444) +} + +fn restore_staged_material(driver: &str, kubernetes_identity: Option<&str>) -> Result<(), String> { + if driver == "kubernetes" { + return delete_kubernetes_config_map( + kubernetes_identity.ok_or("Kubernetes staging identity missing")?, + ); + } + + let source = std::env::var("OPENSHELL_E2E_ADDITIONAL_CA_CERT") + .map_err(|error| format!("additional CA source path missing: {error}"))?; + let artifact = local_staged_artifact()?; + if !std::path::Path::new(&source).is_file() { + return Err(format!("additional CA source does not exist: {source}")); + } + set_staged_artifact_mode(&artifact, 0o644)?; + std::fs::copy(&source, &artifact) + .map_err(|error| format!("restore gateway-owned CA artifact from {source}: {error}"))?; + set_staged_artifact_mode(&artifact, 0o444) +} + +async fn sandbox_id(sandbox_name: &str) -> String { + let (output, code) = run_cli(&["sandbox", "get", sandbox_name, "--output", "json"]).await; + assert_eq!( + code, 0, + "get lifecycle sandbox identity should succeed:\n{output}" + ); + let sandbox: Value = serde_json::from_str(&output).unwrap_or_else(|error| { + panic!("parse lifecycle sandbox identity JSON: {error}; output={output}") + }); + assert_eq!( + sandbox["name"].as_str(), + Some(sandbox_name), + "lifecycle sandbox name changed: {sandbox:#?}" + ); + sandbox["id"] + .as_str() + .unwrap_or_else(|| panic!("lifecycle sandbox identity is missing: {sandbox:#?}")) + .to_string() +} + +async fn assert_kubernetes_invalid_staged_material_fails_closed( + sandbox_name: &str, +) -> Result<(), String> { + let (stop_output, stop_code) = run_cli(&["sandbox", "stop", sandbox_name]).await; + if stop_code != 0 { + return Err(format!( + "stop Kubernetes sandbox before trust validation failed: {stop_output}" + )); + } + + // Ordinary stop/start is the supported reconciliation path. The driver + // must validate the equal-generation immutable ConfigMap before changing + // the Sandbox resource back to an operating state. + let (start_output, start_code) = run_cli(&["sandbox", "start", sandbox_name]).await; + if start_code == 0 { + return Err("Kubernetes sandbox start accepted replaced destination trust".to_string()); + } + let normalized_start_output = start_output + .lines() + .map(|line| line.trim().trim_start_matches('│').trim()) + .collect::>() + .join(" "); + if !normalized_start_output.contains("network additional CA ConfigMap") + || !(normalized_start_output.contains("unexpected trust generation") + || normalized_start_output + .contains("did not retain the exact gateway-normalized trust bundle") + || normalized_start_output.contains("is not immutable") + || normalized_start_output.contains("is not owned by this gateway")) + { + return Err(format!( + "Kubernetes sandbox start lacked stable additional-CA validation evidence: {start_output}" + )); + } + if start_output.contains("-----BEGIN CERTIFICATE-----") + || start_output.contains("-----BEGIN PRIVATE KEY-----") + { + return Err("Kubernetes start validation disclosed CA material".to_string()); + } + Ok(()) +} + +async fn assert_invalid_staged_material_fails_closed(policy_path: &str) { + let startup_error = match SandboxGuard::create(&["--policy", policy_path, "--", "true"]).await { + Ok(mut unexpected) => { + unexpected.cleanup().await; + panic!("a new sandbox accepted invalid staged destination trust"); + } + Err(error) => error, + }; + assert!( + startup_error.contains("error phase") || startup_error.contains("failed"), + "invalid staged trust did not fail sandbox startup: {startup_error}" + ); + // Miette wraps long CLI diagnostics and prefixes continuation lines with a + // box-drawing marker. Normalize that presentation before matching the + // driver's stable error text. + let normalized_startup_error = startup_error + .lines() + .map(|line| line.trim().trim_start_matches('│').trim()) + .collect::>() + .join(" "); + assert!( + normalized_startup_error.contains("--additional-ca-bundle") + || normalized_startup_error.contains("additional destination CA bundle") + || normalized_startup_error.contains("invalid staged destination CA bundle") + || normalized_startup_error + .contains("network additional CA artifact verification failed") + || normalized_startup_error + .contains("does not match the gateway startup trust generation"), + "invalid staged trust lacked stable supervisor additional-CA validation evidence: {startup_error}" + ); + assert!( + !startup_error.contains("-----BEGIN CERTIFICATE-----") + && !startup_error.contains("-----BEGIN PRIVATE KEY-----"), + "invalid staged trust failure disclosed certificate or key material" + ); +} + +fn remove_additional_ca_section(config: &str) -> Result { + const SECTION: &str = "[openshell.supervisor.network]"; + let mut found = false; + let mut skipping = false; + let mut output = Vec::new(); + + for line in config.lines() { + let trimmed = line.trim(); + if trimmed == SECTION { + if found { + return Err("gateway config contains duplicate supervisor network sections".into()); + } + found = true; + skipping = true; + continue; + } + if skipping && trimmed.starts_with('[') { + skipping = false; + } + if !skipping { + output.push(line); + } + } + if !found { + return Err("gateway config does not contain the additional CA section".into()); + } + Ok(format!("{}\n", output.join("\n"))) +} + +async fn restart_kubernetes_gateway_port_forward() -> Result, String> { + // Depending on kubectl and rollout timing, a Service port-forward may + // survive and reconnect to the replacement pod. Keep it when the existing + // registered endpoint is already healthy. + if wait_for_healthy(Duration::from_secs(10)).await.is_ok() { + return Ok(None); + } + + let context = std::env::var("OPENSHELL_E2E_KUBE_CONTEXT_ACTIVE") + .map_err(|error| format!("active Kubernetes context missing: {error}"))?; + let namespace = std::env::var("OPENSHELL_E2E_ADDITIONAL_CA_HELM_NAMESPACE") + .map_err(|error| format!("Helm namespace missing: {error}"))?; + let local_port = std::env::var("OPENSHELL_E2E_KUBE_GATEWAY_LOCAL_PORT") + .map_err(|error| format!("Kubernetes gateway local port missing: {error}"))? + .parse::() + .map_err(|error| format!("Kubernetes gateway local port is invalid: {error}"))?; + + // kubectl port-forward binds to one selected pod even when given a Service. + // A Helm rollout therefore terminates the wrapper-owned process. Wait for + // its local socket to be released, then restore the same endpoint already + // registered in the CLI configuration. + let mut released = false; + for _ in 0..30 { + if let Ok(listener) = TcpListener::bind(("127.0.0.1", local_port)) { + drop(listener); + released = true; + break; + } + std::thread::sleep(Duration::from_secs(1)); + } + if !released { + return Err("previous Kubernetes gateway port-forward did not exit after rollout".into()); + } + + let child = Command::new("kubectl") + .args([ + "--context", + &context, + "-n", + &namespace, + "port-forward", + "svc/openshell", + &format!("{local_port}:8080"), + ]) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .spawn() + .map_err(|error| format!("restart Kubernetes gateway port-forward: {error}"))?; + Ok(Some(PortForwardGuard { child })) +} + +async fn remove_configuration_and_restart( + driver: &str, +) -> Result, String> { + let port_forward = if driver == "kubernetes" { + let context = std::env::var("OPENSHELL_E2E_KUBE_CONTEXT_ACTIVE") + .map_err(|error| format!("active Kubernetes context missing: {error}"))?; + let namespace = std::env::var("OPENSHELL_E2E_ADDITIONAL_CA_HELM_NAMESPACE") + .map_err(|error| format!("Helm namespace missing: {error}"))?; + let release = std::env::var("OPENSHELL_E2E_ADDITIONAL_CA_HELM_RELEASE") + .map_err(|error| format!("Helm release missing: {error}"))?; + let chart = std::env::var("OPENSHELL_E2E_ADDITIONAL_CA_HELM_CHART") + .map_err(|error| format!("Helm chart path missing: {error}"))?; + let output = Command::new("helm") + .args([ + "upgrade", + &release, + &chart, + "--kube-context", + &context, + "--namespace", + &namespace, + "--reuse-values", + "--set-string", + "supervisor.network.additionalCaConfigMapName=", + "--wait", + "--timeout", + "5m", + ]) + .output() + .map_err(|error| format!("remove additional CA Helm value: {error}"))?; + if !output.status.success() { + return Err(format!( + "remove additional CA Helm value failed:\n{}{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + )); + } + restart_kubernetes_gateway_port_forward().await? + } else { + let config_path = std::env::var("OPENSHELL_E2E_GATEWAY_CONFIG") + .map_err(|error| format!("gateway config path missing: {error}"))?; + let config = std::fs::read_to_string(&config_path) + .map_err(|error| format!("read gateway config: {error}"))?; + let without_additional_ca = remove_additional_ca_section(&config)?; + std::fs::write(&config_path, without_additional_ca) + .map_err(|error| format!("write gateway config without additional CA: {error}"))?; + + let gateway = ManagedGateway::from_env()? + .ok_or_else(|| "managed gateway metadata disappeared".to_string())?; + gateway.stop()?; + gateway.start()?; + None + }; + + wait_for_healthy(Duration::from_secs(180)).await?; + Ok(port_forward) +} + +fn replace_kubernetes_config_map_with_invalid_material(identity: &str) -> Result<(), String> { + let mut parts = identity.split('/'); + let namespace = parts.next().ok_or("namespace missing")?; + let config_map = parts.next().ok_or("ConfigMap name missing")?; + let context = std::env::var("OPENSHELL_E2E_KUBE_CONTEXT_ACTIVE") + .map_err(|error| format!("active Kubernetes context missing: {error}"))?; + let current = kubectl_json(&[ + "--context", + &context, + "-n", + namespace, + "get", + "configmap", + config_map, + "-o", + "json", + ])?; + let replacement = serde_json::json!({ + "apiVersion": "v1", + "kind": "ConfigMap", + "metadata": { + "name": config_map, + "namespace": namespace, + "labels": current["metadata"]["labels"].clone(), + "annotations": current["metadata"]["annotations"].clone(), + }, + "immutable": true, + "data": {"ca.crt": "not-a-certificate\n"}, + }); + let mut child = Command::new("kubectl") + .args([ + "--context", + &context, + "-n", + namespace, + "replace", + "--force", + "-f", + "-", + ]) + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .map_err(|error| format!("replace immutable destination CA ConfigMap: {error}"))?; + child + .stdin + .take() + .ok_or("kubectl replacement stdin missing")? + .write_all(replacement.to_string().as_bytes()) + .map_err(|error| format!("write replacement destination CA ConfigMap: {error}"))?; + let output = child + .wait_with_output() + .map_err(|error| format!("wait for destination CA ConfigMap replacement: {error}"))?; + if !output.status.success() { + return Err(format!( + "replace immutable destination CA ConfigMap failed: {}", + String::from_utf8_lossy(&output.stderr) + )); + } + Ok(()) +} + +fn delete_kubernetes_config_map(identity: &str) -> Result<(), String> { + let mut parts = identity.split('/'); + let namespace = parts.next().ok_or("namespace missing")?; + let config_map = parts.next().ok_or("ConfigMap name missing")?; + let context = std::env::var("OPENSHELL_E2E_KUBE_CONTEXT_ACTIVE") + .map_err(|error| format!("active Kubernetes context missing: {error}"))?; + let output = Command::new("kubectl") + .args([ + "--context", + &context, + "-n", + namespace, + "delete", + "configmap", + config_map, + "--wait=true", + ]) + .output() + .map_err(|error| format!("delete invalid destination CA ConfigMap: {error}"))?; + if !output.status.success() { + return Err(format!( + "delete invalid destination CA ConfigMap failed: {}", + String::from_utf8_lossy(&output.stderr) + )); + } + Ok(()) +} + +/// Exec argv that fetches `url` with the workload's default TLS verification +/// and prints the body. Exits nonzero when validation fails, so a failed exec +/// surfaces the verification error. Uses python3 because the Kubernetes and +/// VM workload images do not ship curl. +fn python_fetch_argv(driver: &str, url: &str) -> Vec { + let script = format!( + "import sys, urllib.request\ntry:\n with urllib.request.urlopen({url:?}, timeout=30) as response:\n sys.stdout.write(response.read().decode())\nexcept Exception as error:\n sys.exit(str(error))\n" + ); + vec![ + python_binary_for_driver(driver).to_string(), + "-c".to_string(), + script, + ] +} + +fn python_binary_for_driver(driver: &str) -> &'static str { + match driver { + // The container-engine lanes run the Noble fixture image. + "docker" | "podman" => "/usr/bin/python3", + // The Kubernetes and VM lanes run the uv-based workload image, where + // python3 lives under /usr/local/bin and curl is not present. + _ => "/usr/local/bin/python3", + } +} + +fn hostname_validation_script(matching_host: &str, port: u16) -> String { + format!( + r#" +import json +import socket +import ssl +import sys +import time +import urllib.request + +port = {port} +matching_host = {matching_host:?} +mismatch_host = {MISMATCH_HOST:?} + +# 1. The matching destination must validate through the workload trust +# store (system roots plus the provisioned destination CA). +with urllib.request.urlopen("https://" + matching_host + ":%d/" % port, timeout=30) as response: + body = response.read() +if json.loads(body) != {{"additional_ca": "trusted"}}: + sys.exit("unexpected matching-host body: %r" % (body,)) +print("matching-host-trusted", flush=True) + +# 2. Hostname mismatch against the same reachable fixture: connect to the +# matching host's address while presenting the mismatch hostname as TLS +# SNI. The fixture's private CA is trusted, so the failure must be the +# hostname check itself, not an untrusted issuer or a routing error. +address = socket.getaddrinfo(matching_host, port, proto=socket.IPPROTO_TCP)[0][4] +sock = socket.create_connection(address, timeout=15) +try: + context = ssl.create_default_context() + try: + context.wrap_socket(sock, server_hostname=mismatch_host) + except ssl.SSLCertVerificationError as error: + message = str(error).lower() + if "mismatch" not in message and "doesn't match" not in message: + sys.exit("unexpected hostname verification error: %s" % error) + else: + sys.exit("hostname mismatch was accepted by the destination trust") +finally: + sock.close() +print("hostname-mismatch-rejected", flush=True) + +# 3. Public roots must keep validating alongside the destination CA. +with urllib.request.urlopen("https://example.com/", timeout=30) as response: + response.read() +print("public-root-trusted", flush=True) + +print({READY_MARKER:?}, flush=True) +while True: + time.sleep(1) +"#, + ) +} + +#[tokio::test] +async fn additional_ca_trusts_matching_and_public_hosts_and_rejects_mismatch() { + let driver = driver().expect("select compute driver"); + let port = std::env::var("OPENSHELL_E2E_ADDITIONAL_CA_PORT") + .expect("wrapper must export OPENSHELL_E2E_ADDITIONAL_CA_PORT") + .parse::() + .expect("additional CA fixture port must be numeric"); + let policy = write_policy(port).expect("write additional CA e2e policy"); + let policy_path = policy.path().to_str().expect("policy path must be UTF-8"); + let matching_url = format!("https://host.openshell.internal:{port}/"); + let script = hostname_validation_script("host.openshell.internal", port); + + let mut sandbox = SandboxGuard::create_keep_with_args( + &["--policy", policy_path], + &[python_binary_for_driver(driver), "-c", script.as_str()], + READY_MARKER, + ) + .await + .expect("create sandbox using configured destination CA"); + assert!(sandbox.create_output.contains("matching-host-trusted")); + assert!(sandbox.create_output.contains("hostname-mismatch-rejected")); + assert!(sandbox.create_output.contains("public-root-trusted")); + // Reaching the workload command and then using sandbox exec below requires + // the supervisor's independently configured gateway callback mTLS path. + + let kubernetes_identity = if driver == "kubernetes" { + Some(inspect_kubernetes_pod().expect("inspect Kubernetes ConfigMap and pod placement")) + } else if driver == "vm" { + #[cfg(feature = "e2e-vm")] + { + inspect_vm_overlay().expect("inspect VM overlay and guest-init delivery"); + None + } + #[cfg(not(feature = "e2e-vm"))] + { + panic!("VM driver feature missing"); + } + } else { + #[cfg(any(feature = "e2e-docker", feature = "e2e-podman"))] + { + inspect_local_container(driver, &sandbox.name) + .expect("inspect local container delivery"); + None + } + #[cfg(not(any(feature = "e2e-docker", feature = "e2e-podman")))] + { + panic!("local-container driver feature missing"); + } + }; + + replace_staged_material(driver, kubernetes_identity.as_deref()) + .expect("replace staged destination trust with invalid material"); + if driver == "kubernetes" || driver == "vm" { + let fetch_argv = python_fetch_argv(driver, &matching_url); + let fetch_argv = fetch_argv.iter().map(String::as_str).collect::>(); + let output = sandbox + .exec(&fetch_argv) + .await + .expect("running supervisor retains startup trust after staged material changes"); + assert!( + output.contains(r#"{"additional_ca":"trusted"}"#), + "running supervisor lost startup trust after staged material changed: {output}" + ); + } + + if driver == "kubernetes" { + assert_kubernetes_invalid_staged_material_fails_closed(&sandbox.name) + .await + .expect("stopped Kubernetes sandbox rejects invalid staged trust on start"); + } else if driver == "vm" { + let mut retained_snapshot_sandbox = SandboxGuard::create_keep_with_args( + &["--policy", policy_path], + &[python_binary_for_driver(driver), "-c", script.as_str()], + READY_MARKER, + ) + .await + .expect("VM child uses its retained startup trust snapshot for later launches"); + assert!( + retained_snapshot_sandbox + .create_output + .contains("matching-host-trusted") + ); + retained_snapshot_sandbox.cleanup().await; + } else { + assert_invalid_staged_material_fails_closed(policy_path).await; + } + + // Remove the sandbox whose startup trust and staged material were mutated + // before restoring delivery. For Kubernetes, remove the deliberately + // replaced immutable map so the driver's ensure path can recreate the + // exact gateway startup snapshot for the next sandbox. + sandbox.cleanup().await; + restore_staged_material(driver, kubernetes_identity.as_deref()) + .expect("restore destination trust after invalid-material check"); + + // Use a neutral, long-running main process so stop/start is the only + // lifecycle transition. The exec calls below prove both destination trust + // and gateway-authenticated control traffic independently of that process. + let lifecycle_script = format!("echo {LIFECYCLE_READY_MARKER}; while true; do sleep 1; done"); + let mut lifecycle_sandbox = SandboxGuard::create_keep_with_args( + &["--policy", policy_path], + &["sh", "-lc", &lifecycle_script], + LIFECYCLE_READY_MARKER, + ) + .await + .expect("create lifecycle sandbox using restored destination CA"); + let lifecycle_fetch_argv = python_fetch_argv(driver, &matching_url); + let lifecycle_fetch_argv = lifecycle_fetch_argv + .iter() + .map(String::as_str) + .collect::>(); + let private_ca_output = lifecycle_sandbox + .exec(&lifecycle_fetch_argv) + .await + .expect("lifecycle sandbox trusts the private CA before removal"); + assert!( + private_ca_output.contains(r#"{"additional_ca":"trusted"}"#), + "lifecycle sandbox did not trust private CA before removal: {private_ca_output}" + ); + let durable_marker_output = lifecycle_sandbox + .exec(&[ + "sh", + "-lc", + &format!( + "printf '%s\\n' '{LIFECYCLE_DURABLE_MARKER}' > {LIFECYCLE_DURABLE_PATH}; sync; cat {LIFECYCLE_DURABLE_PATH}" + ), + ]) + .await + .expect("write durable lifecycle marker"); + assert!( + durable_marker_output.contains(LIFECYCLE_DURABLE_MARKER), + "lifecycle sandbox did not write durable marker: {durable_marker_output}" + ); + let lifecycle_id_before_removal = sandbox_id(&lifecycle_sandbox.name).await; + + let (stop_output, stop_code) = run_cli(&["sandbox", "stop", &lifecycle_sandbox.name]).await; + assert_eq!( + stop_code, 0, + "stop lifecycle sandbox should succeed:\n{stop_output}" + ); + wait_for_sandbox_phase(&lifecycle_sandbox.name, "Stopped", Duration::from_secs(180)) + .await + .expect("lifecycle sandbox should be stopped before removing configuration"); + + let _gateway_port_forward = remove_configuration_and_restart(driver) + .await + .expect("remove additional CA configuration and restart gateway"); + + let (start_output, start_code) = run_cli(&["sandbox", "start", &lifecycle_sandbox.name]).await; + assert_eq!( + start_code, 0, + "start the same lifecycle sandbox should succeed:\n{start_output}" + ); + wait_for_sandbox_phase(&lifecycle_sandbox.name, "Ready", Duration::from_secs(180)) + .await + .expect("same lifecycle sandbox should become ready after configuration removal"); + assert_eq!( + sandbox_id(&lifecycle_sandbox.name).await, + lifecycle_id_before_removal, + "stop/start after configuration removal must retain sandbox identity" + ); + + let removed_validation_script = format!( + r#"import os +import sys +import urllib.request + +marker = open({LIFECYCLE_DURABLE_PATH:?}).read().strip() +if marker != {LIFECYCLE_DURABLE_MARKER:?}: + sys.exit("lifecycle durable marker was lost across the restart") +if os.path.exists({ADDITIONAL_CA_MOUNT_PATH:?}): + sys.exit("destination CA mount path survived configuration removal") +try: + urllib.request.urlopen({matching_url:?}, timeout=30).read() +except Exception: + print("same-sandbox-private-ca-rejected", flush=True) +else: + sys.exit("private CA remained trusted after configuration removal") +print("gateway-authenticated-exec-works", flush=True)"# + ); + let removed_validation_argv = [ + python_binary_for_driver(driver).to_string(), + "-c".to_string(), + removed_validation_script, + ]; + let removed_validation_argv = removed_validation_argv + .iter() + .map(String::as_str) + .collect::>(); + let removed_validation_output = lifecycle_sandbox + .exec(&removed_validation_argv) + .await + .expect("gateway-authenticated exec should work after lifecycle restart"); + assert!( + removed_validation_output.contains("gateway-authenticated-exec-works"), + "gateway-authenticated exec was not confirmed: {removed_validation_output}" + ); + assert!( + removed_validation_output.contains("same-sandbox-private-ca-rejected"), + "same sandbox retained private CA after configuration removal: {removed_validation_output}" + ); + + lifecycle_sandbox.cleanup().await; +} diff --git a/e2e/support/gateway-common.sh b/e2e/support/gateway-common.sh index d13d25847..f69518f0b 100644 --- a/e2e/support/gateway-common.sh +++ b/e2e/support/gateway-common.sh @@ -126,6 +126,98 @@ e2e_pick_port() { python3 -c 'import socket; s=socket.socket(); s.bind(("",0)); print(s.getsockname()[1]); s.close()' } +# Generate one ephemeral private destination CA and matching-host HTTPS fixture +# for every compute-driver lane. The caller owns cleanup of the returned PID. +e2e_start_additional_ca_fixture() { + local fixture_dir=$1 + local fixture_log=$2 + local pid_var=$3 + local port_var=$4 + local fixture_pid fixture_port + + mkdir -p "${fixture_dir}" + ( + umask 077 + openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 \ + -out "${fixture_dir}/ca.key" >/dev/null 2>&1 + openssl req -x509 -new -key "${fixture_dir}/ca.key" \ + -out "${fixture_dir}/ca.crt" -days 1 \ + -subj '/CN=OpenShell additional CA e2e' \ + -addext 'basicConstraints=critical,CA:TRUE' \ + -addext 'keyUsage=critical,keyCertSign,cRLSign' >/dev/null 2>&1 + openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 \ + -out "${fixture_dir}/server.key" >/dev/null 2>&1 + openssl req -new -key "${fixture_dir}/server.key" \ + -out "${fixture_dir}/server.csr" \ + -subj '/CN=host.openshell.internal' >/dev/null 2>&1 + cat >"${fixture_dir}/server.ext" <<'EOF' +basicConstraints=critical,CA:FALSE +keyUsage=critical,digitalSignature,keyEncipherment +extendedKeyUsage=serverAuth +subjectAltName=DNS:host.openshell.internal +EOF + openssl x509 -req -in "${fixture_dir}/server.csr" \ + -CA "${fixture_dir}/ca.crt" -CAkey "${fixture_dir}/ca.key" \ + -CAcreateserial -out "${fixture_dir}/server.crt" -days 1 \ + -extfile "${fixture_dir}/server.ext" >/dev/null 2>&1 + ) + + fixture_port="$(e2e_pick_port)" + python3 -u - "${fixture_port}" \ + "${fixture_dir}/server.crt" "${fixture_dir}/server.key" \ + >"${fixture_log}" 2>&1 <<'PY' & +import http.server +import ssl +import sys + +port = int(sys.argv[1]) +context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) +context.load_cert_chain(sys.argv[2], sys.argv[3]) + +class Server(http.server.ThreadingHTTPServer): + def handle_error(self, request, client_address): + # Hostname-mismatch probes intentionally abort during TLS setup. + pass + +class Handler(http.server.BaseHTTPRequestHandler): + def do_GET(self): + body = b'{"additional_ca":"trusted"}' + self.send_response(200) + self.send_header('Content-Type', 'application/json') + self.send_header('Content-Length', str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, fmt, *args): + pass + +server = Server(('0.0.0.0', port), Handler) +server.socket = context.wrap_socket(server.socket, server_side=True) +print('additional-ca-server-listening', flush=True) +server.serve_forever() +PY + fixture_pid=$! + printf -v "${pid_var}" '%s' "${fixture_pid}" + printf -v "${port_var}" '%s' "${fixture_port}" + + local elapsed=0 + while [ "${elapsed}" -lt 30 ]; do + if ! kill -0 "${fixture_pid}" 2>/dev/null; then + echo "ERROR: additional CA HTTPS fixture exited before becoming ready" >&2 + return 1 + fi + if curl -kfsS --max-time 2 "https://127.0.0.1:${fixture_port}/" >/dev/null 2>&1; then + export OPENSHELL_E2E_ADDITIONAL_CA_PORT="${fixture_port}" + export OPENSHELL_E2E_ADDITIONAL_CA_CERT="${fixture_dir}/ca.crt" + return 0 + fi + sleep 1 + elapsed=$((elapsed + 1)) + done + echo "ERROR: additional CA HTTPS fixture did not become ready" >&2 + return 1 +} + e2e_generate_pki() { local gateway_bin=$1 local pki_dir=$2 diff --git a/e2e/with-docker-gateway.sh b/e2e/with-docker-gateway.sh index 47bed1182..290db6b64 100755 --- a/e2e/with-docker-gateway.sh +++ b/e2e/with-docker-gateway.sh @@ -126,6 +126,11 @@ DOCKER_NETWORK_MANAGED=0 GPU_MODE="${OPENSHELL_E2E_DOCKER_GPU:-0}" OIDC_MODE="${OPENSHELL_E2E_OIDC_GATEWAY:-0}" OIDC_ISSUER="${OPENSHELL_E2E_OIDC_ISSUER:-}" +ADDITIONAL_CA_MODE="${OPENSHELL_E2E_ADDITIONAL_CA:-0}" +ADDITIONAL_CA_SERVER_PID="" +ADDITIONAL_CA_SERVER_LOG="${WORKDIR}/additional-ca-server.log" +ADDITIONAL_CA_DIR="${WORKDIR}/additional-ca" +ADDITIONAL_CA_PORT="" if [ "${OIDC_MODE}" = "1" ] && [ -z "${OIDC_ISSUER}" ]; then echo "ERROR: OPENSHELL_E2E_OIDC_ISSUER is required when OPENSHELL_E2E_OIDC_GATEWAY=1" >&2 @@ -145,6 +150,7 @@ cleanup() { e2e_stop_gateway "${GATEWAY_PID}" "${GATEWAY_PID_FILE}" e2e_stop_process "${DRIVER_PID}" "external Docker compute driver" + e2e_stop_process "${ADDITIONAL_CA_SERVER_PID}" "additional CA HTTPS fixture" if [ "${exit_code}" -ne 0 ] \ && [ -n "${E2E_NAMESPACE}" ] \ @@ -198,6 +204,11 @@ cleanup() { cat "${DRIVER_LOG}" || true echo "=== end external Docker compute driver log ===" fi + if [ "${exit_code}" -ne 0 ] && [ -f "${ADDITIONAL_CA_SERVER_LOG}" ]; then + echo "=== additional CA HTTPS fixture log ===" + cat "${ADDITIONAL_CA_SERVER_LOG}" || true + echo "=== end additional CA HTTPS fixture log ===" + fi rm -rf "${WORKDIR}" 2>/dev/null || true } @@ -268,6 +279,10 @@ connect_current_container_to_docker_network() { } if [ -n "${OPENSHELL_GATEWAY_ENDPOINT:-}" ]; then + if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then + echo "ERROR: additional CA e2e requires the wrapper-managed Docker gateway." >&2 + exit 2 + fi case "${OPENSHELL_GATEWAY_ENDPOINT}" in http://*) ;; https://*) @@ -544,6 +559,19 @@ if ! ensure_sandbox_image_available "${SANDBOX_IMAGE}"; then exit 2 fi +start_additional_ca_fixture() { + e2e_start_additional_ca_fixture \ + "${ADDITIONAL_CA_DIR}" "${ADDITIONAL_CA_SERVER_LOG}" \ + ADDITIONAL_CA_SERVER_PID ADDITIONAL_CA_PORT +} +if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then + if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then + echo "ERROR: additional CA e2e requires the in-process Docker driver." >&2 + exit 2 + fi + start_additional_ca_fixture +fi + HOST_PORT=$(e2e_pick_port) HEALTH_PORT=$(e2e_pick_port) STATE_DIR="${XDG_STATE_HOME}" @@ -561,6 +589,10 @@ export OPENSHELL_E2E_DOCKER_NETWORK_NAME="${DOCKER_NETWORK_NAME}" export OPENSHELL_E2E_NETWORK_NAME="${DOCKER_NETWORK_NAME}" export OPENSHELL_E2E_SANDBOX_NAMESPACE="${E2E_NAMESPACE}" export OPENSHELL_E2E_DRIVER="docker" +if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then + export OPENSHELL_E2E_ADDITIONAL_CA_ARTIFACT="${XDG_STATE_HOME}/openshell/network-supervisor" + export OPENSHELL_E2E_GATEWAY_CONFIG="${STATE_DIR}/gateway.toml" +fi if connect_current_container_to_docker_network "${DOCKER_NETWORK_NAME}"; then echo "Connected CI job container to Docker network ${DOCKER_NETWORK_NAME} (${GATEWAY_HOST_ALIAS_IP})." # Container jobs use the host Docker daemon. The host-networked supervisor @@ -613,6 +645,11 @@ GATEWAY_CONFIG="${STATE_DIR}/gateway.toml" e2e_write_gateway_oidc_config "${OPENSHELL_OIDC_ISSUER}" fi fi + if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then + printf '[openshell.supervisor.network]\n' + printf 'additional_ca_bundle = %s\n\n' \ + "$(toml_string "${ADDITIONAL_CA_DIR}/ca.crt")" + fi printf '[openshell.drivers.docker]\n' if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then printf 'socket_path = %s\n' "$(toml_string "${DRIVER_SOCKET}")" diff --git a/e2e/with-kube-gateway.sh b/e2e/with-kube-gateway.sh index 13e58b10b..7d05c9a5c 100755 --- a/e2e/with-kube-gateway.sh +++ b/e2e/with-kube-gateway.sh @@ -128,6 +128,12 @@ OPENSHIFT_ROUTE_HOST="" # Temp dir holding the client mTLS material extracted from openshell-client-tls # for the OpenShift Route transport. Removed by cleanup(). OPENSHIFT_PKI_DIR="${WORKDIR}/openshift-pki" +ADDITIONAL_CA_MODE="${OPENSHELL_E2E_ADDITIONAL_CA:-0}" +ADDITIONAL_CA_SOURCE_CONFIG_MAP="openshell-e2e-additional-ca" +ADDITIONAL_CA_SERVER_PID="" +ADDITIONAL_CA_SERVER_LOG="${WORKDIR}/additional-ca-server.log" +ADDITIONAL_CA_DIR="${WORKDIR}/additional-ca" +ADDITIONAL_CA_SOURCE_CREATED=0 # Isolate CLI/SDK gateway metadata from the developer's real config. export XDG_CONFIG_HOME="${WORKDIR}/config" @@ -510,6 +516,15 @@ cleanup() { local exit_code=$? stop_gateway_portforward + e2e_stop_process "${ADDITIONAL_CA_SERVER_PID}" "additional CA HTTPS fixture" + + if [ "${ADDITIONAL_CA_SOURCE_CREATED}" = "1" ] \ + && [ -n "${KUBE_CONTEXT}" ] && [ -n "${NAMESPACE}" ] \ + && command -v kubectl >/dev/null 2>&1; then + kctl -n "${NAMESPACE}" delete configmap "${ADDITIONAL_CA_SOURCE_CONFIG_MAP}" \ + --ignore-not-found >/dev/null 2>&1 || true + ADDITIONAL_CA_SOURCE_CREATED=0 + fi if [ "${exit_code}" -ne 0 ] && [ -n "${KUBE_CONTEXT}" ] && [ -n "${NAMESPACE}" ]; then if command -v kubectl >/dev/null 2>&1 \ @@ -763,6 +778,12 @@ run_scenario() { export OPENSHELL_E2E_KUBE_CHART="${ROOT}/deploy/helm/openshell" export OPENSHELL_E2E_KUBECONFIG="${KUBECONFIG:-}" export OPENSHELL_PROVISION_TIMEOUT="${OPENSHELL_PROVISION_TIMEOUT:-300}" + if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then + export OPENSHELL_E2E_ADDITIONAL_CA_HELM_NAMESPACE="${NAMESPACE}" + export OPENSHELL_E2E_ADDITIONAL_CA_HELM_RELEASE="${RELEASE_NAME}" + export OPENSHELL_E2E_ADDITIONAL_CA_HELM_CHART="${ROOT}/deploy/helm/openshell" + export OPENSHELL_E2E_KUBE_GATEWAY_LOCAL_PORT="${LOCAL_PORT:-}" + fi e2e_import_example_provider_profiles \ "${OPENSHELL_BIN:-${ROOT}/target/debug/openshell}" "${ROOT}" || return 1 @@ -1224,6 +1245,29 @@ if kctl api-resources --api-group=route.openshift.io --no-headers 2>/dev/null | fi fi +start_additional_ca_fixture() { + [ "${ADDITIONAL_CA_MODE}" = "1" ] || return 0 + if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then + echo "ERROR: Kubernetes additional CA e2e requires the in-process Kubernetes driver." >&2 + return 1 + fi + if [ -z "${HOST_GATEWAY_IP}" ]; then + echo "ERROR: Kubernetes additional CA e2e requires a host gateway IP for host.openshell.internal." >&2 + return 1 + fi + + e2e_start_additional_ca_fixture \ + "${ADDITIONAL_CA_DIR}" "${ADDITIONAL_CA_SERVER_LOG}" \ + ADDITIONAL_CA_SERVER_PID ADDITIONAL_CA_PORT + kctl create namespace "${NAMESPACE}" --dry-run=client -o yaml | kctl apply -f - + kctl -n "${NAMESPACE}" create configmap "${ADDITIONAL_CA_SOURCE_CONFIG_MAP}" \ + --from-file="ca.crt=${ADDITIONAL_CA_DIR}/ca.crt" \ + --dry-run=client -o yaml | kctl apply -f - + ADDITIONAL_CA_SOURCE_CREATED=1 +} + +start_additional_ca_fixture + ACTIVE_CREDENTIAL_DRIVER="${OPENSHELL_E2E_CREDENTIAL_DRIVER:-kubernetes-secrets}" if [ "${OPENSHELL_E2E_CREDENTIAL_DRIVERS:-0}" = "1" ] \ && [ "${ACTIVE_CREDENTIAL_DRIVER}" = "vault" ]; then @@ -1234,6 +1278,9 @@ helm_extra_args=() helm_post_renderer_args=() helm_extra_args+=(--set "server.telemetryEnabled=${OPENSHELL_TELEMETRY_ENABLED}") helm_extra_args+=(--set supervisor.sandboxRuntime.networkPolicyEnforced=true) +if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then + helm_extra_args+=(--set "supervisor.network.additionalCaConfigMapName=${ADDITIONAL_CA_SOURCE_CONFIG_MAP}") +fi if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then if [ "${OPENSHELL_E2E_KUBE_BUILD_IMAGES}" != "1" ]; then echo "ERROR: external Kubernetes driver e2e requires OPENSHELL_E2E_KUBE_BUILD_IMAGES=1." >&2 @@ -1527,6 +1574,12 @@ else export OPENSHELL_E2E_KUBE_CHART="${ROOT}/deploy/helm/openshell" export OPENSHELL_E2E_KUBECONFIG="${KUBECONFIG:-}" export OPENSHELL_PROVISION_TIMEOUT="${OPENSHELL_PROVISION_TIMEOUT:-300}" + if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then + export OPENSHELL_E2E_ADDITIONAL_CA_HELM_NAMESPACE="${NAMESPACE}" + export OPENSHELL_E2E_ADDITIONAL_CA_HELM_RELEASE="${RELEASE_NAME}" + export OPENSHELL_E2E_ADDITIONAL_CA_HELM_CHART="${ROOT}/deploy/helm/openshell" + export OPENSHELL_E2E_KUBE_GATEWAY_LOCAL_PORT="${LOCAL_PORT:-}" + fi e2e_import_example_provider_profiles \ "${OPENSHELL_BIN:-${ROOT}/target/debug/openshell}" "${ROOT}" || exit 1 diff --git a/e2e/with-podman-gateway.sh b/e2e/with-podman-gateway.sh index 57edfa0a4..224eecaaf 100755 --- a/e2e/with-podman-gateway.sh +++ b/e2e/with-podman-gateway.sh @@ -131,6 +131,11 @@ PODMAN_SOCKET="" GPU_MODE="${OPENSHELL_E2E_PODMAN_GPU:-0}" OIDC_MODE="${OPENSHELL_E2E_OIDC_GATEWAY:-0}" OIDC_ISSUER="${OPENSHELL_E2E_OIDC_ISSUER:-}" +ADDITIONAL_CA_MODE="${OPENSHELL_E2E_ADDITIONAL_CA:-0}" +ADDITIONAL_CA_SERVER_PID="" +ADDITIONAL_CA_SERVER_LOG="${WORKDIR}/additional-ca-server.log" +ADDITIONAL_CA_DIR="${WORKDIR}/additional-ca" +ADDITIONAL_CA_PORT="" if [ "${OIDC_MODE}" = "1" ] && [ -z "${OIDC_ISSUER}" ]; then echo "ERROR: OPENSHELL_E2E_OIDC_ISSUER is required when OPENSHELL_E2E_OIDC_GATEWAY=1" >&2 @@ -145,6 +150,7 @@ cleanup() { e2e_stop_gateway "${GATEWAY_PID}" "${GATEWAY_PID_FILE}" e2e_stop_process "${DRIVER_PID}" "external Podman compute driver" + e2e_stop_process "${ADDITIONAL_CA_SERVER_PID}" "additional CA HTTPS fixture" local sandbox_ids="" if command -v podman >/dev/null 2>&1; then @@ -215,6 +221,11 @@ cleanup() { cat "${PODMAN_SERVICE_LOG}" || true echo "=== end podman service log ===" fi + if [ "${exit_code}" -ne 0 ] && [ -f "${ADDITIONAL_CA_SERVER_LOG}" ]; then + echo "=== additional CA HTTPS fixture log ===" + cat "${ADDITIONAL_CA_SERVER_LOG}" || true + echo "=== end additional CA HTTPS fixture log ===" + fi if [ -n "${PODMAN_SERVICE_PID}" ]; then kill "${PODMAN_SERVICE_PID}" >/dev/null 2>&1 || true @@ -441,6 +452,10 @@ ensure_podman_sandbox_runtime_image() { } if [ -n "${OPENSHELL_GATEWAY_ENDPOINT:-}" ]; then + if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then + echo "ERROR: additional CA e2e requires the wrapper-managed Podman gateway." >&2 + exit 2 + fi case "${OPENSHELL_GATEWAY_ENDPOINT}" in http://*) ;; https://*) @@ -515,6 +530,19 @@ PKI_DIR="${WORKDIR}/pki" e2e_generate_pki "${GATEWAY_BIN}" "${PKI_DIR}" "host.containers.internal" export OPENSHELL_E2E_GATEWAY_CA_CERT="${PKI_DIR}/ca.crt" +start_additional_ca_fixture() { + e2e_start_additional_ca_fixture \ + "${ADDITIONAL_CA_DIR}" "${ADDITIONAL_CA_SERVER_LOG}" \ + ADDITIONAL_CA_SERVER_PID ADDITIONAL_CA_PORT +} +if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then + if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then + echo "ERROR: additional CA e2e requires the in-process Podman driver." >&2 + exit 2 + fi + start_additional_ca_fixture +fi + HOST_PORT=$(e2e_pick_port) HEALTH_PORT=$(e2e_pick_port) PRIMARY_BIND_IP="127.0.0.1" @@ -532,6 +560,10 @@ ensure_e2e_podman_network "${PODMAN_NETWORK_NAME}" export OPENSHELL_E2E_DRIVER="podman" export OPENSHELL_E2E_NETWORK_NAME="${PODMAN_NETWORK_NAME}" export OPENSHELL_E2E_SANDBOX_NAMESPACE="${E2E_NAMESPACE}" +if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then + export OPENSHELL_E2E_ADDITIONAL_CA_ARTIFACT="${XDG_STATE_HOME}/openshell/network-supervisor" + export OPENSHELL_E2E_GATEWAY_CONFIG="${STATE_DIR}/gateway.toml" +fi echo "Starting openshell-gateway on port ${HOST_PORT} (namespace: ${E2E_NAMESPACE})..." e2e_generate_gateway_jwt "${JWT_DIR}" @@ -555,6 +587,20 @@ e2e_write_podman_gateway_config \ "${OPENSHELL_PODMAN_SOCKET:-}" \ "${OIDC_MODE}" \ "${OPENSHELL_OIDC_ISSUER:-}" +if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then + CONFIG_WITH_ADDITIONAL_CA="${GATEWAY_CONFIG}.additional-ca" + awk -v ca_path="$(e2e_podman_toml_string "${ADDITIONAL_CA_DIR}/ca.crt")" ' + !inserted && $0 == "[openshell.drivers.podman]" { + print "[openshell.supervisor.network]" + print "additional_ca_bundle = " ca_path + print "" + inserted = 1 + } + { print } + END { if (!inserted) exit 42 } + ' "${GATEWAY_CONFIG}" >"${CONFIG_WITH_ADDITIONAL_CA}" + mv "${CONFIG_WITH_ADDITIONAL_CA}" "${GATEWAY_CONFIG}" +fi EXTERNAL_DRIVER_GRPC_ENDPOINT="https://127.0.0.1:${HOST_PORT}" EXTERNAL_DRIVER_HEALTH_CHECK_INTERVAL_SECS=10 EXTERNAL_DRIVER_ENABLE_BIND_MOUNTS=true diff --git a/examples/governance-interceptor/Cargo.lock b/examples/governance-interceptor/Cargo.lock index 11a44da02..71f25e0db 100644 --- a/examples/governance-interceptor/Cargo.lock +++ b/examples/governance-interceptor/Cargo.lock @@ -1860,6 +1860,7 @@ dependencies = [ "base64 0.22.1", "glob", "ipnet", + "libc", "miette", "nix", "openshell-crypto", diff --git a/examples/isolation-backend-v8/Cargo.lock b/examples/isolation-backend-v8/Cargo.lock index a0bdcf165..b86eb5a6e 100644 --- a/examples/isolation-backend-v8/Cargo.lock +++ b/examples/isolation-backend-v8/Cargo.lock @@ -2129,6 +2129,7 @@ dependencies = [ "glob", "ipnet", "jsonwebtoken", + "libc", "miette", "nix", "openshell-crypto", diff --git a/examples/supervisor-middleware-content-guard/Cargo.lock b/examples/supervisor-middleware-content-guard/Cargo.lock index cf2c9ed99..9b7d4bc5f 100644 --- a/examples/supervisor-middleware-content-guard/Cargo.lock +++ b/examples/supervisor-middleware-content-guard/Cargo.lock @@ -2014,6 +2014,7 @@ dependencies = [ "base64 0.22.1", "glob", "ipnet", + "libc", "miette", "nix", "openshell-crypto", diff --git a/proto/compute_driver.proto b/proto/compute_driver.proto index 65da87689..2eba7ef2a 100644 --- a/proto/compute_driver.proto +++ b/proto/compute_driver.proto @@ -480,6 +480,12 @@ message StartSandboxRequest { bool resume = 7; // Forwarded to ResumeBoundaryRequest.restart_if_restore_fails. bool restart_if_restore_fails = 8; + // Durable provisioning snapshot recovered by the gateway from its persisted + // Sandbox record. Local drivers use this only to reconcile a stopped + // workload/supervisor pair without trusting mutable runtime inspection + // output. Older gateways omit the field, preserving the original start + // contract on the wire. + optional DriverSandbox sandbox = 9; } message SuspendSandboxRequest { diff --git a/rfc/0003-gateway-configuration/README.md b/rfc/0003-gateway-configuration/README.md index 6e42e7665..530280fbd 100644 --- a/rfc/0003-gateway-configuration/README.md +++ b/rfc/0003-gateway-configuration/README.md @@ -96,6 +96,11 @@ disable_tls = false # Gateway-owned CA injected into the selected local driver for supervisor TLS. guest_tls_ca = "/etc/openshell/certs/ca.pem" +# Shared sandbox destination trust. These gateway-local paths are read and +# normalized at startup; the resulting material is delivered by in-tree drivers. +[openshell.supervisor.network] +additional_ca_bundle = "/etc/openshell/certs/private-destination-ca.pem" + [openshell.gateway.tls] cert_path = "/etc/openshell/certs/gateway.pem" key_path = "/etc/openshell/certs/gateway-key.pem" @@ -172,8 +177,9 @@ Driver authors define and own their config schema. Adding a new driver does not Field-level merge rules: 1. **`[openshell.gateway]`** populates `openshell_core::Config` (including the nested `[openshell.gateway.tls]` and `[openshell.gateway.oidc]` tables, which map to `TlsConfig` and `OidcConfig` respectively). -2. **`[openshell.drivers.]`** is propagated to the driver crate, which deserializes it into its own struct. Driver schemas evolve independently of the gateway's core `Config`. -3. **CLI / env** override any value set by steps 1–2, field by field. The override check uses clap's `ValueSource` — a value is applied from the file only when the corresponding flag was not supplied via the command line or environment. +2. **`[openshell.supervisor.network]`** holds shared network-supervisor settings. `additional_ca_bundle` names a single gateway-local certificate-only PEM file. Gateway startup strictly normalizes it before an in-tree compute driver receives the material. The setting augments sandbox destination trust and does not configure proxy, listener, OIDC, or gateway callback trust. +3. **`[openshell.drivers.]`** is propagated to the driver crate, which deserializes it into its own struct. Driver schemas evolve independently of the gateway's core `Config`. +4. **CLI / env** override any value set by steps 1–3, field by field. The override check uses clap's `ValueSource` — a value is applied from the file only when the corresponding flag was not supplied via the command line or environment. `bind_address`, `health_bind_address`, and `metrics_bind_address` are stored as `SocketAddr` (IP + port). The CLI exposes them as a single `--bind-address` IP plus `--port`, `--health-port`, and `--metrics-port`; CLI overrides apply to the matching part of the parsed `SocketAddr`. @@ -203,6 +209,7 @@ The following cross-field validations are applied after merging file + env + CLI - Gateway listener TLS requires `cert_path` and `key_path`; `client_ca_path` is required only for listener client-certificate verification. TLS-enabled Docker, Podman, and VM drivers also require a complete gateway-owned guest CA, certificate, and key bundle. Kubernetes projects guest TLS through a Secret instead. - `database_url` must be non-empty after merging env + CLI — every supported driver requires it. The field is not accepted from the file (see Secrets above). - `compute_driver` selects exactly one driver. When omitted, the gateway falls back to auto-detection. A custom driver requires a named table with `socket_path`, unless startup supplies an explicit socket override. The legacy `compute_drivers` list is rejected. +- `additional_ca_bundle` must name a readable, non-empty certificate-only PEM file with at least one usable certificate. Private keys, unrelated PEM items, DER-only input, inline PEM, and URLs are rejected. Configured material is unsupported with remote or custom compute drivers. ### Schema compatibility diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index b90cdce41..485ce9c29 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -427,13 +427,13 @@ Common findings: probe and the companion supervisor's private health check. Do not add capabilities, attach a workload network, or disable the runtime seccomp profile. There is no sandbox nftables or nested-network setup to repair. -- Gateway fails to start with an `additional_ca_bundle` error: the Podman - driver validates the setting at startup. The file must exist on the gateway - host, be non-empty, and contain at least one PEM certificate. +- Gateway fails to start with an `additional_ca_bundle` error: gateway startup + validates the configured single bundle. The file must exist on the gateway + host, be a regular non-symlink file, and contain at least one PEM certificate. - Inspected HTTPS to a private-CA destination fails (for example `curl: (52) Empty reply from server`), or the supervisor exits at startup, - with `additional_ca_bundle` set: the supervisor container needs the CA, not - just the workload. Check + with a global `additional_ca_bundle` configured: the supervisor container + needs the staged CA, not just the workload. Check `podman inspect -supervisor --format '{{json .Mounts}}'` for a read-only bind of the bundle at `/etc/openshell/tls/additional/ca-bundle.pem`, and `--format '{{json .Config.Cmd}}'` for `--additional-ca-bundle` with that @@ -523,6 +523,92 @@ kubectl -n openshell logs -c openshell-gateway --tail=200 `server.telemetryEnabled` renders `OPENSHELL_TELEMETRY_ENABLED` on the gateway pod, and the gateway propagates the effective value to sandbox supervisors. +#### Additional destination CA ConfigMap + +When `supervisor.network.additionalCaConfigMapName` is set, Helm must mount the +**source** operator ConfigMap's `ca.crt` key into the gateway at +`/etc/openshell-tls/additional-ca-source/ca.crt`. The rendered +`gateway.toml` must contain the global (not Kubernetes-driver-local) setting: + +```toml +[openshell.supervisor.network] +additional_ca_bundle = "/etc/openshell-tls/additional-ca-source/ca.crt" +``` + +Inspect the source, render, and live gateway mount before inspecting sandbox +pods: + +```bash +helm -n openshell get values openshell | grep -A3 additionalCaConfigMapName +kubectl -n openshell get configmap -o jsonpath='{.data.ca\.crt}' >/dev/null +kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -A2 '^\[openshell.supervisor.network\]' +kubectl -n openshell get pod -o jsonpath='{.spec.volumes[?(@.name=="additional-ca-source")]}{"\n"}' +``` + +At startup, the gateway normalizes the source once and stages a +content-addressed, read-only artifact below +`/var/lib/openshell-state/openshell/network-supervisor/`. The gateway workload +sets `XDG_STATE_HOME=/var/lib/openshell-state` and mounts the writable +`gateway-state` volume there. Check that wiring before debugging the driver: + +```bash +kubectl -n openshell get pod -o jsonpath='{.spec.containers[0].env[?(@.name=="XDG_STATE_HOME")]}{"\n"}' +kubectl -n openshell get pod -o jsonpath='{.spec.containers[0].volumeMounts[?(@.name=="gateway-state")]}{"\n"}' +kubectl -n openshell get pod -o jsonpath='{.spec.volumes[?(@.name=="gateway-state")]}{"\n"}' +``` + +At sandbox creation or restart, the Kubernetes driver creates or verifies an +immutable, content-addressed ConfigMap in the target sandbox namespace. Its +name binds the effective gateway identity and normalized trust digest. The +managed ConfigMap must have `openshell.ai/managed-by=openshell`, the matching +`openshell.ai/gateway-id` label, and an +`openshell.ai/additional-ca-generation` annotation. Different trust +generations intentionally coexist so running sandboxes retain their startup +trust. The supervisor Pod mounts the ConfigMap read-only at +`/etc/openshell/tls/additional/ca-bundle.pem` and receives +`--additional-ca-bundle` for that path; the immutable flag plus the driver's +per-creation digest and exact-byte validation close deletion/recreation races +after the API check. + +Kubernetes RBAC cannot restrict `create` by `resourceNames` or restrict `get` +by a name prefix. When this feature is enabled, the chart grants the gateway +service account ConfigMap `get`/`create` in the shared sandbox namespace. In +managed/operator modes, the existing ClusterRoleBinding makes those permissions +cluster-wide. It does not grant ConfigMap `list`, `watch`, `patch`, `update`, or +`delete`. Prefer a dedicated gateway service account and namespace-scoped +bindings when the read boundary matters. Check the objects and authorization: + +```bash +kubectl -n get configmap \ + -l openshell.ai/managed-by=openshell,openshell.ai/gateway-id= \ + --show-labels +kubectl auth can-i get configmaps -n \ + --as system:serviceaccount:openshell: +kubectl auth can-i create configmaps -n \ + --as system:serviceaccount:openshell: +kubectl auth can-i patch configmaps -n \ + --as system:serviceaccount:openshell: +kubectl auth can-i delete configmaps -n \ + --as system:serviceaccount:openshell: +``` + +A missing, mutable, unowned, wrong-generation, or wrong-data managed ConfigMap +fails closed. Do not relabel or edit it blindly. Investigate the object owner; +remove an invalid object only after confirming no sandbox uses it, then retry +so the driver can recreate the exact generation. Diagnostics deliberately omit +certificate contents; inspect the source object separately rather than copying +PEM into logs or tickets. + +Changing source ConfigMap contents does not reload the gateway or a running +supervisor: restart the gateway to reread/normalize it, then recreate or +restart affected sandboxes. Removing the Helm setting means newly created or +recreated sandbox pods have no additional-CA volume/mount, but existing running +pods retain startup trust until restarted. The driver intentionally has no +ConfigMap list/delete permission; unused managed generations can remain. After +all sandboxes are migrated, use an operator identity to verify the generation +annotation and pod references, then explicitly delete unused ConfigMaps in each +applicable namespace. Do not add list/delete RBAC as a cleanup workaround. + When no external credential driver is enabled, the Helm chart uses the gateway's default encrypted database credential storage. The chart creates a retained Kubernetes Secret for the shared KEK, injects it into gateway pods, and diff --git a/tests/testkit/Cargo.lock b/tests/testkit/Cargo.lock index 3cd07a667..92db6b2b1 100644 --- a/tests/testkit/Cargo.lock +++ b/tests/testkit/Cargo.lock @@ -1888,6 +1888,7 @@ dependencies = [ "chrono", "glob", "ipnet", + "libc", "miette", "nix", "openshell-crypto",