trending/OpenShell
1
0
Fork 0
mirror of https://github.com/NVIDIA/OpenShell.git synced 2026-10-29 07:23:25 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat(network): support additional destination CAs (#3292)

Browse Source 
* feat(network): support additional destination CAs

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(network): harden additional CA propagation

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(network): narrow additional CA review scope

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(network): harden additional CA rotation

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* feat(network): complete additional CA delivery

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(docker): remove stale runtime fixture field

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(network): adapt additional CA series to upstream main

Resolve rebase fallout against current upstream main after the series was
rebased 214 commits forward:

- follow upstream #3110: supervisor→gateway TLS is CA-only, so
  configure_gateway_tls and its isolation tests drop the client identity
- follow upstream tls_secret_names reduction to [String; 1] in the Podman
  driver while keeping the network trust bundle delivery
- port [openshell.supervisor.network] rendering into the
  openshell.gatewayConfigToml serializer in _toml.tpl
- adapt tests to upstream struct/function changes (DriverStartupContext,
  SandboxRunConfig, IsolationSpecInput, supervisor_pod, VM bootstrap image)
- drop conformance-artifact-prefix plumbing removed upstream

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* refactor(network): adopt upstream additional CA bundle surface

Unify the additional destination CA series on the mechanism merged to
main in #3996: one operator pre-assembled PEM bundle, the
--additional-ca-bundle supervisor flag, and the
/etc/openshell/tls/additional/ca-bundle.pem staging path.

- Replace the gateway-level additional_ca_cert_paths list with a single
  additional_ca_bundle file validated and staged at gateway startup.
- Restore the upstream supervisor argument path and fold-into-roots
  behavior; drop the --network-additional-ca-bundle and
  --network-additional-ca-digest contracts.
- Keep gateway normalization bounds, content-addressed staging,
  digest-based driver reconciliation, and stop/start rotation; the
  freshness check now runs at the gateway/driver boundary instead of
  the supervisor boundary.
- Reject additional_ca_bundle in operator driver JSON config; standalone
  drivers self-source the bundle from --sandbox-additional-ca-bundle
  with the same validation.
- Extend the Kubernetes ConfigMap name digest segment to keep the full
  63-byte budget after the prefix rename.
- Box the persisted-sandbox reconciliation futures so callers stay
  under the clippy::large_futures threshold.

Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and
tests, e2e wiring, gateway configuration docs, and the cluster
debugging skill to the single mechanism.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(ci): make gateway startup tests wrap-tolerant and refresh example lockfile

The network trust startup tests asserted on raw substrings of gateway
diagnostics, but the CLI renderer wraps output at a fixed width with
continuation markers and hard-breaks long paths. Compare assertions on
whitespace- and continuation-squashed text so they hold regardless of
wrap position.

Format the e2e workspace (separate cargo workspace, not covered by the
root cargo fmt run) and refresh examples/isolation-backend-v8/Cargo.lock
for the libc dependency added to the network trust crate.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(podman): reuse TLS CA secret during stopped-pair trust replacement

The additional CA e2e never ran in CI before the first maintainer test
dispatch, and its local container inspection was broken in two ways: it
located the supervisor container through e2e network membership (the
supervisor is host-networked or runtime-bridged, so the filter matched
the CI job container instead), and it required the gateway mTLS CA as a
bind mount (Podman delivers it as a container secret when user
namespaces remap UIDs). Select the supervisor by managed labels and
accept the secret delivery form.

The lifecycle leg then exposed a driver bug: stopped-pair trust
replacement re-created the per-sandbox TLS CA secret, colliding with
the secret still owned by the renamed stopped pair. Mirror the token
and resolver secret handling: require and reuse the existing secret
during replacement instead of creating it.

Validated end-to-end with the Podman additional CA e2e lane.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(server): restore upstream recovery semantics for start-request failures

Recovery no longer marks sandboxes Error/InvalidPersistedSandbox when
driver_start_request fails during the startup or lifecycle sweeps.
Gateway configuration problems (for example a missing launch signer)
must not permanently fail unrelated sandbox records; the sweep logs a
warning and skips them, matching the behavior before the additional CA
work. Sandbox-record failures surface through the normal start path.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* refactor(podman): refresh stale supervisors without replacing workloads

Replace the stopped-pair trust reconciliation with supervisor-only
recreation. The destination-trust generation label moves from the
workload to the supervisor container, and start_sandbox recreates only
the supervisor when its generation is missing or stale. The workload
container, its volumes, its secrets, and its workspace data are never
replaced: the additional CA bundle is mounted only into the supervisor.

Running sandboxes are never gated on trust labels, and recovery without
fresh launch credentials keeps the existing supervisor. The intentional
-removal fence machinery in the watcher existed only for workload
replacement and is removed with it.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* refactor(docker): drop workload replacement for destination-trust changes

Delete the stopped-workload trust reconciliation path and all of its
machinery: the trust-generation label on workload containers, the
generation match check in the start fence, the stopped-sandbox
replacement, the bounded workspace archive copy with its 512 MiB cap
and regular-file-only validation, and the boundary-state rollback
helpers.

No replacement machinery is needed: the trusted supervisor container
is already re-created from the gateway's current configuration on
every start via ensure_control_process_for_container, so stop/start
picks up the current additional CA bundle while the workload
container, its volumes, and its workspace data are never replaced.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* docs(configuration): drop reconciliation wording from additional CA lifecycle

The stop/start lifecycle description already matches the
supervisor-only refresh model; remove the two remaining references to
restart reconciliation of workload containers.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(network): port additional CA test fixtures to the crypto interface

Route the gateway TLS isolation test PKI, the destination CA source
fixtures, and the gateway startup trust tests through
openshell_crypto::pki now that rcgen access is centralized there:
generate_keypair/self_signed/signed_by replace the direct rcgen calls
and CertifiedKey field access follows the wrapper's shape.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(e2e): repair additional CA lane for Docker and non-container builds

The first maintainer test dispatch revealed two latent issues in the
additional CA e2e that no local run had exercised: the Podman gateway
CA secret helper referenced ContainerEngine without the container
feature gate, breaking Kubernetes and VM builds of the test, and the
Docker branch asserted a client ca.crt bind mount that the Docker
driver never creates. Gate the helper like its callers, and assert
Docker's actual delivery path instead: the gateway mTLS CA staged as
tls/ca.pem inside the read-only supervisor state volume, selected
through the OPENSHELL_TLS_CA environment pointer.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(helm): restore alphabetical README ordering for supervisor keys

The helm:docs check regenerates the chart README with helm-docs, which
orders rows alphabetically; a conflict resolution had placed
supervisor.network.additionalCaConfigMapName after the
supervisor.podSecurityContext rows. Regenerate with mise run
helm:docs.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* chore: refresh testkit lockfile for openshell-core libc dependency

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(driver-podman): stage operator CA bundle as driver-owned artifact

Standalone Podman drivers accepting --sandbox-additional-ca-bundle
previously kept the operator's source path as the trust artifact path.
The immutable-artifact verification performed on every create/start
requires exactly mode 0444, which an operator-owned source path cannot
guarantee, so a readable-but-writable source failed reconciliation.

Read the operator bundle with the same bounded, no-symlink validation,
then stage the normalized material into a driver-owned artifact inside a
private tempdir held for the driver process lifetime. The artifact is
created 0600, synced, and only then restricted to 0444 so no observer
ever sees a writable artifact. The operator source and its permissions
are never referenced after startup.

Addresses GATOR-291f43b2-02.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(driver-podman): pin retained workload image during supervisor refresh

Supervisor-only trust reconciliation re-entered
create_sandbox_with_existing_token, which re-pulled and re-resolved the
workload image tag even though the workload container is retained. If
the tag moved to a different image between stop and start, identity
resolution and child-environment construction would reflect the new
image while the retained workload still runs the original one, leaving
mode-0600 bootstrap files the retained user cannot read.

During a refresh with a retained workload, inspect the stopped container
and pin to the immutable image ID it was created from (Config.Image);
never pull or resolve the request tag. The requested reference is still
used for child metadata so the retained workload's environment is
byte-identical to its original launch.

Addresses GATOR-291f43b2-01.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): verify host-only VM trust delivery and drop curl dependency

The additional-CA lane's VM assertions read the destination CA out of the
guest overlay and a guest-side supervisor-argv marker — artifacts from
the guest-supervisor design that no longer exist now that the VM driver
stages the CA host-only and runs the supervisor on the host. Inspect the
driver-staged host artifact (digest-compared against the source), assert
the guest overlay contains no CA material at the old mount paths, and
keep the serial-console no-disclosure check.

The workload validation scripts also ran curl, which exists only in the
container-engine fixture image; the Kubernetes and VM lanes use the
uv-based workload image and died instantly with "canonical main process
already finished". Rewrite the main-process, exec, and lifecycle
validation scripts in python3 (present in every lane's workload image).
The network-policy binary identity is the resolved /proc/<pid>/exe
target, so the versioned python3.12 binaries are allowlisted rather
than the unversioned symlinks. The hostname mismatch check keeps its
semantics: same reachable fixture, mismatched TLS SNI, and the failure
must specifically be hostname verification.

Addresses GATOR-291f43b2-04.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(kubernetes): allow admission-added labels on destination-trust ConfigMaps

The destination-trust ConfigMap validation compared the whole label map
for equality. A mutating admission webhook that attaches an
organizational label to the created ConfigMap made the create response
fail validation, and the retained ConfigMap then failed every subsequent
create/start through the existing-object path, blocking sandbox launches
despite the ownership labels, immutable flag, digest, and certificate
bytes remaining correct.

Validate the gateway-owned label pairs individually and allow unrelated
labels; the exact trust-material checks are unchanged. Covered by a
regression test with an admission-added label and a missing
gateway-owned label.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(podman): close supervisor-refresh rollback gaps

The supervisor-only trust refresh renamed the previous supervisor to a
random rollback name and then performed the token-secret lookup. A
transient lookup failure returned with the supervisor still renamed,
where a normal delete could never find it, and the rollback rename and
final rollback removal ignored their own failures, leaving orphans with
attached control and channel resources.

Complete every fallible preparation before the rename. On a failed
replacement, restore the canonical name; if the restore itself fails,
return an error naming the rollback container so the orphan stays
identifiable — a later retry still succeeds because the canonical name
is absent and the replacement path recreates a fresh supervisor. Treat
not-found during the final rollback removal as success and name the
orphan in both the log and the error on other removal failures; a retry
observes the fresh canonical supervisor and resumes the normal start
path.

Covered by failure-injection tests for each transaction boundary:
preparation failure before any rename, replacement failure restoring
the canonical name, and restore failure naming the orphaned rollback
container.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* docs: address destination-trust review corrections

The gateway-configuration RFC documented an additional_ca_cert_paths
list that the strict configuration parser never accepted; document the
additional_ca_bundle single-path setting the parser actually enforces.

The debug-openshell-cluster skill named the wrong generation annotation
(openshell.ai/network-supervisor-trust-generation) and described digest
argv and network-init canonicalization checks the implementation does
not perform. Correct it to openshell.ai/additional-ca-generation and
describe the actual delivery: the supervisor Pod mounts the immutable
ConfigMap read-only and receives --additional-ca-bundle for the mount
path, with the driver validating the digest annotation and exact bytes
per creation.

Classify the destination-trust generation hashes as deterministic
identity derivation rather than authenticity proofs, following the
openshell-crypto exclusions table: trust decisions come from the
gateway's normalization, immutable staging, exact-byte validation, and
TLS verification. Record the classification in the crypto README and at
each hashing site.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(network): reconcile additional CA tests with suspend/resume and auth changes

Upstream's suspend/resume lifecycle took StartSandboxRequest fields 7
and 8, so the durable-snapshot field moves from 7 to 9; update the
wire-format test to assert the field-9 key (0x4a) and initialize the
new resume controls. Repair the cli test module after the auth opt-in
test merged alongside the environment-guard helpers, and initialize the
gateway driver-startup context's network-trust fields in the shared
supervision test.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(server): box start-recovery futures for clippy large-futures

The durable-snapshot field grows StartSandboxRequest enough that the
start-recovery match and the recovery-expiry test exceed the
large-futures limit under -D warnings. Box::pin both futures, matching
the pattern the recovery path used before the suspend/resume rework.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

---------

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
This commit is contained in:
Jesse Jaggars
2026-10-27 18:31:11 +00:00
committed by GitHub
parent 26134dc735
commit 2c31ac7e60
71 changed files with 8024 additions and 434 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/branch-e2e.yml
+37 -1
View File
@@ -328,6 +328,19 @@ jobs:
suite-matrix: >-
[{"suite":"ci","runner":"ubuntu-26.04","podman_major":"5","podman_package_version":"5.7.0+ds2-3build1","conmon_package_version":"2.1.13+ds1-2","cmd":"mise run --no-deps --skip-deps e2e:podman:ci"}]
podman-additional-ca-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-podman-test.yml
with:
image-tag: ${{ github.sha }}
suite-matrix: >-
[{"suite":"additional-ca","runner":"ubuntu-26.04","podman_major":"5","podman_package_version":"5.7.0+ds2-3build1","conmon_package_version":"2.1.13+ds1-2","cmd":"OPENSHELL_E2E_PODMAN_TEST=additional_ca mise run --no-deps --skip-deps e2e:podman"}]
vm-e2e:
needs: [pr_metadata, build-binaries, build-vm-driver]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
@@ -337,6 +350,18 @@ jobs:
packages: read
uses: ./.github/workflows/e2e-vm-test.yml
vm-additional-ca-e2e:
needs: [pr_metadata, build-binaries, build-vm-driver]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-vm-test.yml
with:
suite-name: additional-ca
test-name: additional_ca
docker-external-driver-e2e:
needs: [pr_metadata, build-binaries, build-gateway-plain, build-external-drivers, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
@@ -401,8 +426,16 @@ jobs:
include:
- agent_sandbox_api: v1beta1
agent_sandbox_version: v1.0.3
job_name: Kubernetes E2E (Rust smoke, Agent Sandbox v1beta1)
test_name: ""
- agent_sandbox_api: v1alpha1
agent_sandbox_version: v0.4.6
job_name: Kubernetes E2E (Rust smoke, Agent Sandbox v1alpha1)
test_name: ""
- agent_sandbox_api: v1beta1
agent_sandbox_version: v0.5.0
job_name: Kubernetes E2E (additional destination CA, Agent Sandbox v1beta1)
test_name: additional_ca
permissions:
actions: read
contents: read
@@ -410,8 +443,9 @@ jobs:
uses: ./.github/workflows/e2e-kubernetes-test.yml
with:
image-tag: ${{ github.sha }}
job-name: Kubernetes E2E (Rust smoke, Agent Sandbox ${{ matrix.agent_sandbox_api }})
job-name: ${{ matrix.job_name }}
agent-sandbox-version: ${{ matrix.agent_sandbox_version }}
test-name: ${{ matrix.test_name }}
kubernetes-workspace-managed-e2e:
needs: [pr_metadata, build-binaries, build-images]
@@ -494,7 +528,9 @@ jobs:
- driver-specific-integration
- docker-e2e
- podman-e2e
- podman-additional-ca-e2e
- vm-e2e
- vm-additional-ca-e2e
- docker-external-driver-e2e
- podman-external-driver-e2e
- vm-external-driver-e2e
.github/workflows/e2e-docker-test.yml
+1 -1
View File
@@ -29,7 +29,7 @@ on:
required: false
type: string
default: >-
[{"suite":"python","cmd":"mise run --no-deps --skip-deps e2e:python","apt_packages":"","python_proto":true,"mcp":false},{"suite":"oidc-python","cmd":"mise run --no-deps --skip-deps e2e:oidc-python:docker","apt_packages":"","python_proto":true,"mcp":false},{"suite":"oidc-pkce-docker","cmd":"mise run --no-deps --skip-deps e2e:oidc-pkce:docker","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"rust-docker","cmd":"mise run --no-deps --skip-deps e2e:rust","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"rust-docker-push","cmd":"mise run --no-deps --skip-deps e2e:rust:push","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"oci-image","cmd":"mise run --no-deps --skip-deps e2e:oci-image","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"mcp","cmd":"mise run --no-deps --skip-deps e2e:mcp","apt_packages":"","python_proto":false,"mcp":true}]
[{"suite":"python","cmd":"mise run --no-deps --skip-deps e2e:python","apt_packages":"","python_proto":true,"mcp":false},{"suite":"oidc-python","cmd":"mise run --no-deps --skip-deps e2e:oidc-python:docker","apt_packages":"","python_proto":true,"mcp":false},{"suite":"oidc-pkce-docker","cmd":"mise run --no-deps --skip-deps e2e:oidc-pkce:docker","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"rust-docker","cmd":"mise run --no-deps --skip-deps e2e:rust","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"rust-docker-push","cmd":"mise run --no-deps --skip-deps e2e:rust:push","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"oci-image","cmd":"mise run --no-deps --skip-deps e2e:oci-image","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"additional-ca","cmd":"OPENSHELL_E2E_DOCKER_TEST=additional_ca mise run --no-deps --skip-deps e2e:docker","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"mcp","cmd":"mise run --no-deps --skip-deps e2e:mcp","apt_packages":"","python_proto":false,"mcp":true}]
permissions:
actions: read
.github/workflows/e2e-vm-test.yml
+5
View File
@@ -22,6 +22,10 @@ on:
required: false
type: string
default: e2e:vm
test-name:
required: false
type: string
default: ""
permissions:
actions: read
@@ -55,4 +59,5 @@ jobs:
- name: Run tests
env:
E2E_TASK: ${{ inputs.e2e-task }}
OPENSHELL_E2E_VM_TEST: ${{ inputs.test-name }}
run: mise run --no-deps --skip-deps "$E2E_TASK"
Cargo.lock
Generated
+7
View File
@@ -4372,8 +4372,10 @@ dependencies = [
"base64 0.22.1",
"chrono",
"glob",
"h2",
"ipnet",
"jsonwebtoken",
"libc",
"miette",
"nix 0.29.0",
"openshell-crypto",
@@ -4514,6 +4516,7 @@ dependencies = [
"rustls",
"serde",
"serde_json",
"sha2 0.10.9",
"temp-env",
"thiserror 2.0.20",
"tokio",
@@ -4595,6 +4598,7 @@ dependencies = [
"rustix 1.1.4",
"serde",
"serde_json",
"sha2 0.10.9",
"tar",
"temp-env",
"tempfile",
@@ -4665,6 +4669,7 @@ dependencies = [
"polling",
"prost",
"prost-types",
"rcgen",
"rustix 1.1.4",
"serde",
"serde_json",
@@ -4710,6 +4715,7 @@ dependencies = [
"miette",
"nix 0.29.0",
"openshell-core",
"openshell-crypto",
"openshell-driver-docker",
"openshell-driver-kubernetes",
"openshell-driver-mxc",
@@ -4717,6 +4723,7 @@ dependencies = [
"openshell-otel",
"openshell-policy",
"openshell-server",
"rcgen",
"rustix 1.1.4",
"serde",
"serde_json",
crates/openshell-core/Cargo.toml
+3 -1
View File
@@ -46,6 +46,7 @@ openshell-otel = { path = "../openshell-otel", optional = true }
[target.'cfg(unix)'.dependencies]
nix = { workspace = true }
rustix = { workspace = true }
libc = "0.2"
[features]
default = ["telemetry"]
@@ -66,9 +67,10 @@ tonic-prost-build = { workspace = true }
protoc-bin-vendored = { workspace = true }
[dev-dependencies]
h2 = "0.4"
rcgen = { workspace = true }
tempfile = "3"
tracing-subscriber = { workspace = true }
rcgen = { workspace = true }
tokio-rustls = { workspace = true }
[lints]
crates/openshell-core/src/driver_utils.rs
+1 -1
View File
@@ -545,7 +545,7 @@ fn read_regular_file_bounded(path: &str, max_bytes: u64) -> Result<String, Bound
use std::os::unix::fs::OpenOptionsExt as _;
std::fs::OpenOptions::new()
.read(true)
.custom_flags(nix::libc::O_NONBLOCK)
.custom_flags(libc::O_NONBLOCK)
.open(path)
};
#[cfg(not(unix))]
crates/openshell-core/src/grpc_client.rs
+143 -21
View File
@@ -370,6 +370,24 @@ impl tonic::service::Interceptor for AuthInterceptor {
}
}
/// Configure server-authenticated TLS for the gateway control-plane endpoint.
///
/// Trust only the configured gateway CA. Destination, system, and proxy
/// trust are intentionally not parameters to this control-plane boundary.
fn configure_gateway_tls(
ep: Endpoint,
ca_pem: Vec<u8>,
server_name: Option<&str>,
) -> Result<Endpoint> {
let mut tls_config = ClientTlsConfig::new().ca_certificate(Certificate::from_pem(ca_pem));
if let Some(server_name) = server_name.filter(|name| !name.is_empty()) {
tls_config = tls_config.domain_name(server_name);
}
ep.tls_config(tls_config)
.into_diagnostic()
.wrap_err("failed to configure TLS")
}
/// Build the plain (un-intercepted) gRPC channel.
///
/// When the endpoint uses `https://`, server-authenticated TLS is configured
@@ -405,27 +423,12 @@ async fn build_plain_channel(endpoint: &str) -> Result<Channel> {
.into_diagnostic()
.wrap_err_with(|| format!("failed to read CA cert from {ca_path}"))?;
// Trust only the configured CA — this is the chart's internal CA
// that signs the gateway's internal server certificate. The gateway uses SNI-based
// certificate selection to present this internal cert to supervisor
// connections, so no public root trust is needed here.
//
// Do NOT add `.with_native_roots()` or `.with_webpki_roots()` here:
// the supervisor runs inside the user-selected sandbox image
// (Docker/Podman drivers), and broadening the trust store would let
// an attacker who controls the image + DNS present a publicly valid
// certificate and intercept the supervisor→gateway TLS connection.
let mut tls_config = ClientTlsConfig::new().ca_certificate(Certificate::from_pem(ca_pem));
if let Ok(server_name) = std::env::var(sandbox_env::GATEWAY_TLS_SERVER_NAME)
&& !server_name.is_empty()
{
tls_config = tls_config.domain_name(server_name);
}
ep = ep
.tls_config(tls_config)
.into_diagnostic()
.wrap_err("failed to configure TLS")?;
// The gateway uses SNI-based certificate selection to present its
// internal cert to supervisor connections. Do not add native,
// webpki, destination, or child-process roots here: an image that can
// influence DNS must not broaden supervisor→gateway authentication.
let server_name = std::env::var(sandbox_env::GATEWAY_TLS_SERVER_NAME).ok();
ep = configure_gateway_tls(ep, ca_pem, server_name.as_deref())?;
}
ep.connect()
@@ -462,6 +465,125 @@ impl Drop for ClientSpanStatus {
}
}
#[cfg(test)]
mod gateway_tls_isolation_tests {
use super::*;
use rcgen::{CertificateParams, IsCa};
use rustls::pki_types::{CertificateDer, PrivateKeyDer};
use std::io::Cursor;
use tokio::net::TcpListener;
use tokio_rustls::TlsAcceptor;
#[allow(clippy::struct_field_names)]
struct TestPki {
ca_pem: Vec<u8>,
server_cert_pem: Vec<u8>,
server_key_pem: Vec<u8>,
}
fn test_pki(server_name: &str) -> TestPki {
let mut ca_params = CertificateParams::new(Vec::<String>::new()).unwrap();
ca_params.is_ca = IsCa::Ca(rcgen::BasicConstraints::Unconstrained);
ca_params.key_usages = vec![
rcgen::KeyUsagePurpose::KeyCertSign,
rcgen::KeyUsagePurpose::CrlSign,
];
let ca_key = openshell_crypto::pki::generate_keypair().unwrap();
let ca = openshell_crypto::pki::self_signed(ca_params, &ca_key).unwrap();
let server_key = openshell_crypto::pki::generate_keypair().unwrap();
let server_params = CertificateParams::new(vec![server_name.to_string()]).unwrap();
let server =
openshell_crypto::pki::signed_by(server_params, &server_key, &ca, &ca_key).unwrap();
TestPki {
ca_pem: ca.pem().into_bytes(),
server_cert_pem: server.pem().into_bytes(),
server_key_pem: server_key.serialize_pem().unwrap().into_bytes(),
}
}
async fn start_h2_tls_server(pki: &TestPki) -> std::net::SocketAddr {
let certificates = rustls_pemfile::certs(&mut Cursor::new(&pki.server_cert_pem))
.collect::<std::result::Result<Vec<CertificateDer<'static>>, _>>()
.unwrap();
let key: PrivateKeyDer<'static> =
rustls_pemfile::private_key(&mut Cursor::new(&pki.server_key_pem))
.unwrap()
.unwrap();
let mut config = rustls::ServerConfig::builder()
.with_no_client_auth()
.with_single_cert(certificates, key)
.unwrap();
config.alpn_protocols = vec![b"h2".to_vec()];
let acceptor = TlsAcceptor::from(Arc::new(config));
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
tokio::spawn(async move {
let Ok((stream, _)) = listener.accept().await else {
return;
};
let Ok(stream) = acceptor.accept(stream).await else {
return;
};
if let Ok(mut connection) = h2::server::handshake(stream).await {
while let Some(request) = connection.accept().await {
if request.is_err() {
break;
}
}
}
});
address
}
async fn connect_with_gateway_ca(
address: std::net::SocketAddr,
pki: &TestPki,
server_name: &str,
) -> Result<Channel> {
let endpoint = Endpoint::from_shared(format!("https://{address}")).into_diagnostic()?;
configure_gateway_tls(endpoint, pki.ca_pem.clone(), Some(server_name))?
.connect()
.await
.into_diagnostic()
}
#[tokio::test]
async fn destination_ca_cannot_authenticate_gateway() {
const GATEWAY_NAME: &str = "gateway.internal.test";
let gateway = test_pki(GATEWAY_NAME);
let destination = test_pki(GATEWAY_NAME);
let real_address = start_h2_tls_server(&gateway).await;
tokio::time::timeout(
Duration::from_secs(5),
connect_with_gateway_ca(real_address, &gateway, GATEWAY_NAME),
)
.await
.expect("real gateway connection timed out")
.expect("configured gateway CA should authenticate the gateway");
let fake_address = start_h2_tls_server(&destination).await;
let error = tokio::time::timeout(
Duration::from_secs(5),
connect_with_gateway_ca(fake_address, &gateway, GATEWAY_NAME),
)
.await
.expect("fake gateway rejection timed out")
.expect_err("destination-signed fake gateway must be rejected");
assert!(format!("{error:?}").contains("certificate"), "{error:?}");
let mismatch_address = start_h2_tls_server(&gateway).await;
tokio::time::timeout(
Duration::from_secs(5),
connect_with_gateway_ca(mismatch_address, &gateway, "wrong.internal.test"),
)
.await
.expect("hostname mismatch rejection timed out")
.expect_err("normal server-name verification must remain enabled");
}
}
/// Build a Bearer-authenticated channel to the gateway with one sandbox
/// instance's credentials. The first connection starts that instance's
/// renewal task.
crates/openshell-core/src/lib.rs
+2
View File
@@ -40,6 +40,7 @@ pub mod mcp;
pub mod metadata;
pub mod middleware;
pub mod net;
pub mod network_trust;
#[cfg(feature = "oauth")]
pub mod oauth;
pub mod paths;
@@ -79,6 +80,7 @@ pub use error::{ComputeDriverError, Error, Result};
pub use metadata::{
GetResourceVersion, ObjectId, ObjectLabels, ObjectName, ObjectWorkspace, SetResourceVersion,
};
pub use network_trust::NetworkSupervisorTrustBundle;
pub use sandbox_session::{SandboxSessionId, SandboxSessionIdError};
/// Build version string derived from git metadata.
crates/openshell-core/src/network_trust.rs
+427
View File
@@ -0,0 +1,427 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
//! Shared representation of gateway-configured destination trust material.
//!
//! The gateway owns validation and creation of this material. This type keeps
//! the normalized certificate bytes separate from gateway mTLS material while
//! giving compute-driver adapters read-only access to the data they must stage
//! for the network supervisor.
use std::fmt;
use std::path::{Path, PathBuf};
use std::sync::Arc;
/// Maximum number of normalized PEM bytes accepted for destination trust.
///
/// This bound is shared by the gateway normalization boundary and all compute
/// driver consumers. Kubernetes counts the `ca.crt` key and value toward its
/// 1 MiB `ConfigMap` data limit, so reserve the six key bytes here to keep every
/// supported driver on the same deployable boundary.
pub const MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES: usize = 1024 * 1024 - "ca.crt".len();
/// Protected resource metadata key recording the destination-trust generation
/// used to build a sandbox startup artifact.
///
/// Kubernetes stores this value in an annotation because a `sha256:<hex>`
/// digest exceeds the 63-character label-value limit. Local container drivers
/// may store the same protected key in a container label.
pub const NETWORK_SUPERVISOR_TRUST_GENERATION_KEY: &str = "openshell.ai/additional-ca-generation";
/// Explicit generation marker for an omitted destination-trust configuration.
///
/// An explicit value, rather than an absent marker, lets a stopped sandbox
/// distinguish a removal from a resource created by an older gateway.
pub const NETWORK_SUPERVISOR_TRUST_GENERATION_NONE: &str = "none";
/// Normalized, gateway-owned trust material for sandbox destination TLS.
///
/// The PEM bytes contain only canonical X.509 certificate blocks. The type is
/// deliberately distinct from gateway client TLS material: its contents must
/// never be used to authenticate the gateway control plane.
#[derive(Clone, Eq, PartialEq)]
pub struct NetworkSupervisorTrustBundle {
normalized_pem: Arc<[u8]>,
certificate_count: usize,
digest: String,
artifact_path: PathBuf,
}
impl NetworkSupervisorTrustBundle {
/// Construct a bundle from already-normalized certificate material.
///
/// Callers are responsible for validating and canonicalizing the PEM
/// before constructing the shared value. The gateway performs that work
/// at startup; drivers only consume the resulting read-only data.
#[must_use]
pub fn new(
normalized_pem: Vec<u8>,
certificate_count: usize,
digest: impl Into<String>,
artifact_path: PathBuf,
) -> Self {
Self {
normalized_pem: Arc::from(normalized_pem),
certificate_count,
digest: digest.into(),
artifact_path,
}
}
/// Return the canonical certificate PEM bytes.
#[must_use]
pub fn pem(&self) -> &[u8] {
&self.normalized_pem
}
/// Alias documenting that the returned bytes are normalized material.
#[must_use]
pub fn normalized_pem(&self) -> &[u8] {
self.pem()
}
/// Return the number of certificates in the normalized bundle.
#[must_use]
pub const fn certificate_count(&self) -> usize {
self.certificate_count
}
/// Return the non-secret digest of the normalized bundle.
#[must_use]
pub fn digest(&self) -> &str {
&self.digest
}
/// Return the gateway-owned host path that local/VM adapters may stage.
#[must_use]
pub fn artifact_path(&self) -> &Path {
&self.artifact_path
}
/// Return whether the bundle contains no normalized bytes.
#[must_use]
pub fn is_empty(&self) -> bool {
self.normalized_pem.is_empty()
}
/// Verify that the staged artifact still exactly matches this startup
/// snapshot.
///
/// Local drivers call this before building or replacing a stopped sandbox.
/// The check is deliberately bounded and follows no symlinks on Unix so a
/// changed, missing, or non-regular artifact fails closed before a runtime
/// can consume it. Diagnostics identify only the artifact path and digest;
/// certificate contents are never included.
pub fn verify_artifact(&self) -> Result<(), NetworkSupervisorTrustArtifactError> {
use std::fs::{self, OpenOptions};
use std::io::Read as _;
let path = self.artifact_path();
let metadata = fs::symlink_metadata(path).map_err(|error| {
NetworkSupervisorTrustArtifactError::Read {
path: path.to_path_buf(),
error,
}
})?;
if !metadata.file_type().is_file() {
return Err(NetworkSupervisorTrustArtifactError::NotRegularFile {
path: path.to_path_buf(),
});
}
if !has_expected_artifact_permissions(&metadata) {
return Err(NetworkSupervisorTrustArtifactError::InsecurePermissions {
path: path.to_path_buf(),
});
}
if metadata.len() > MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES as u64 {
return Err(NetworkSupervisorTrustArtifactError::TooLarge {
path: path.to_path_buf(),
limit: MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES,
});
}
let mut options = OpenOptions::new();
options.read(true);
#[cfg(unix)]
{
use std::os::unix::fs::OpenOptionsExt as _;
options.custom_flags(libc::O_CLOEXEC | libc::O_NOFOLLOW | libc::O_NONBLOCK);
}
let mut file =
options
.open(path)
.map_err(|error| NetworkSupervisorTrustArtifactError::Read {
path: path.to_path_buf(),
error,
})?;
let opened_metadata =
file.metadata()
.map_err(|error| NetworkSupervisorTrustArtifactError::Read {
path: path.to_path_buf(),
error,
})?;
if !opened_metadata.file_type().is_file() {
return Err(NetworkSupervisorTrustArtifactError::NotRegularFile {
path: path.to_path_buf(),
});
}
if !has_expected_artifact_permissions(&opened_metadata) {
return Err(NetworkSupervisorTrustArtifactError::InsecurePermissions {
path: path.to_path_buf(),
});
}
if opened_metadata.len() > MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES as u64 {
return Err(NetworkSupervisorTrustArtifactError::TooLarge {
path: path.to_path_buf(),
limit: MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES,
});
}
let capacity = usize::try_from(opened_metadata.len()).map_err(|_| {
NetworkSupervisorTrustArtifactError::TooLarge {
path: path.to_path_buf(),
limit: MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES,
}
})?;
let mut actual = Vec::with_capacity(capacity);
file.by_ref()
.take((MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES + 1) as u64)
.read_to_end(&mut actual)
.map_err(|error| NetworkSupervisorTrustArtifactError::Read {
path: path.to_path_buf(),
error,
})?;
if actual.len() > MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES {
return Err(NetworkSupervisorTrustArtifactError::TooLarge {
path: path.to_path_buf(),
limit: MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES,
});
}
if actual != self.normalized_pem() {
return Err(NetworkSupervisorTrustArtifactError::GenerationMismatch {
path: path.to_path_buf(),
digest: self.digest.clone(),
});
}
Ok(())
}
}
/// Return whether metadata has the read-only artifact permissions created by
/// the gateway. The containing `0700` directory limits access to its owner;
/// `0444` allows both root Docker and rootless Podman consumers to read a bind
/// mounted generation without allowing any process to modify it.
fn has_expected_artifact_permissions(metadata: &std::fs::Metadata) -> bool {
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt as _;
metadata.permissions().mode() & 0o7777 == 0o444
}
#[cfg(not(unix))]
{
let _ = metadata;
true
}
}
/// Non-secret failures while re-reading a gateway-owned staged trust artifact.
#[derive(Debug, thiserror::Error)]
pub enum NetworkSupervisorTrustArtifactError {
#[error("network additional CA artifact '{path}' could not be read: {error}")]
Read {
path: PathBuf,
#[source]
error: std::io::Error,
},
#[error("network additional CA artifact '{path}' is not a regular file")]
NotRegularFile { path: PathBuf },
#[error(
"network additional CA artifact '{path}' does not have the required read-only permissions"
)]
InsecurePermissions { path: PathBuf },
#[error("network additional CA artifact '{path}' exceeds the shared {limit}-byte limit")]
TooLarge { path: PathBuf, limit: usize },
#[error(
"network additional CA artifact '{path}' does not match the gateway startup trust generation {digest}"
)]
GenerationMismatch { path: PathBuf, digest: String },
}
impl fmt::Debug for NetworkSupervisorTrustBundle {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("NetworkSupervisorTrustBundle")
.field("certificate_count", &self.certificate_count)
.field("digest", &self.digest)
.field("artifact_path", &self.artifact_path)
.finish_non_exhaustive()
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn debug_representation_redacts_pem_bytes() {
let pem = b"-----BEGIN CERTIFICATE-----\nnot-secret-in-a-real-fixture\n-----END CERTIFICATE-----\n";
let bundle = NetworkSupervisorTrustBundle::new(
pem.to_vec(),
1,
"sha256:test-digest",
PathBuf::from("/var/lib/openshell/network/additional-ca.crt"),
);
let debug = format!("{bundle:?}");
assert!(debug.contains("certificate_count: 1"));
assert!(debug.contains("sha256:test-digest"));
assert!(debug.contains("additional-ca.crt"));
assert!(!debug.contains("not-secret-in-a-real-fixture"));
assert_eq!(bundle.pem(), pem);
assert_eq!(bundle.normalized_pem(), pem);
assert_eq!(bundle.certificate_count(), 1);
assert_eq!(bundle.digest(), "sha256:test-digest");
assert_eq!(
bundle.artifact_path(),
Path::new("/var/lib/openshell/network/additional-ca.crt")
);
assert!(!bundle.is_empty());
}
#[test]
fn cloned_bundles_share_immutable_material() {
let bundle = NetworkSupervisorTrustBundle::new(
b"certificate".to_vec(),
1,
"sha256:test",
PathBuf::from("/tmp/additional-ca.crt"),
);
assert_eq!(bundle.clone(), bundle);
}
fn test_bundle(path: PathBuf, contents: &[u8]) -> NetworkSupervisorTrustBundle {
NetworkSupervisorTrustBundle::new(contents.to_vec(), 1, "sha256:test-generation", path)
}
fn set_read_only(path: &Path) {
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt as _;
std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o444)).unwrap();
}
#[cfg(not(unix))]
let _ = path;
}
#[cfg(unix)]
#[test]
fn artifact_verifier_rejects_writable_files() {
use std::os::unix::fs::PermissionsExt as _;
let temp = tempfile::tempdir().unwrap();
let path = temp.path().join("additional-ca.crt");
std::fs::write(&path, b"normalized test CA\n").unwrap();
std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o644)).unwrap();
let error = test_bundle(path, b"normalized test CA\n")
.verify_artifact()
.unwrap_err();
assert!(matches!(
error,
NetworkSupervisorTrustArtifactError::InsecurePermissions { .. }
));
assert!(!error.to_string().contains("normalized test CA"));
}
#[test]
fn artifact_verifier_accepts_exact_file_contents() {
let temp = tempfile::tempdir().unwrap();
let path = temp.path().join("additional-ca.crt");
let contents = b"normalized test CA\n";
std::fs::write(&path, contents).unwrap();
set_read_only(&path);
test_bundle(path, contents).verify_artifact().unwrap();
}
#[test]
fn artifact_verifier_rejects_replaced_contents_without_disclosing_them() {
let temp = tempfile::tempdir().unwrap();
let path = temp.path().join("additional-ca.crt");
let expected = b"expected normalized CA\n";
let replacement = b"replacement private CA must not be disclosed\n";
std::fs::write(&path, replacement).unwrap();
set_read_only(&path);
let error = test_bundle(path, expected).verify_artifact().unwrap_err();
assert!(matches!(
error,
NetworkSupervisorTrustArtifactError::GenerationMismatch { .. }
));
let diagnostic = error.to_string();
assert!(diagnostic.contains("sha256:test-generation"));
assert!(!diagnostic.contains("expected normalized CA"));
assert!(!diagnostic.contains("replacement private CA"));
}
#[test]
fn artifact_verifier_rejects_non_regular_files() {
let temp = tempfile::tempdir().unwrap();
let path = temp.path().join("directory");
std::fs::create_dir(&path).unwrap();
let error = test_bundle(path, b"contents")
.verify_artifact()
.unwrap_err();
assert!(matches!(
error,
NetworkSupervisorTrustArtifactError::NotRegularFile { .. }
));
}
#[cfg(unix)]
#[test]
fn artifact_verifier_rejects_symlinks() {
use std::os::unix::fs::symlink;
let temp = tempfile::tempdir().unwrap();
let target = temp.path().join("target");
let link = temp.path().join("additional-ca.crt");
std::fs::write(&target, b"normalized test CA\n").unwrap();
symlink(&target, &link).unwrap();
let error = test_bundle(link, b"normalized test CA\n")
.verify_artifact()
.unwrap_err();
assert!(matches!(
error,
NetworkSupervisorTrustArtifactError::NotRegularFile { .. }
));
}
#[test]
fn artifact_verifier_enforces_shared_size_bound() {
let temp = tempfile::tempdir().unwrap();
let path = temp.path().join("additional-ca.crt");
let at_limit = vec![b'x'; MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES];
std::fs::write(&path, &at_limit).unwrap();
set_read_only(&path);
test_bundle(path.clone(), &at_limit)
.verify_artifact()
.unwrap();
let oversized = vec![b'x'; MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES + 1];
std::fs::remove_file(&path).unwrap();
std::fs::write(&path, &oversized).unwrap();
set_read_only(&path);
let error = test_bundle(path, b"expected")
.verify_artifact()
.unwrap_err();
assert!(matches!(
error,
NetworkSupervisorTrustArtifactError::TooLarge { limit, .. }
if limit == MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES
));
assert!(!error.to_string().contains(&"x".repeat(64)));
}
}
crates/openshell-crypto/README.md
+1
View File
@@ -140,6 +140,7 @@ not a whole-application crypto coverage claim:
| Provider-refresh advisory-lock keys | Deterministic database lock selection for coordinating refreshes, not authentication or encryption. These remain outside the selected-backend contract. |
| Endpoint IDs and pagination fingerprints | Deterministic identification/request matching, not authentication. These remain outside the selected-backend contract. |
| Debug token fingerprints | Diagnostic display only; direct hashing remains outside the contract even though the input is a credential. |
| Destination-trust generation labels (`openshell-server` network trust, Podman operator staging, Kubernetes ConfigMap naming, VM staged-snapshot checks) | Deterministic identity derivation for naming and generation equality, not encryption or proof of authorization. Trust decisions use the gateway's normalization, immutable staging, exact-byte validation, and TLS verification rather than these digests. These retain `sha2` directly. |
| Schema compatibility tests, content-cache identities, retry jitter, test fixtures | Non-authentication uses remain unchanged. |
Cargo feature selection does not prevent a crate from adding another crypto
crates/openshell-driver-docker/src/lib.rs
+139 -36
View File
@@ -58,7 +58,8 @@ use openshell_core::proto_struct::{
deserialize_optional_non_empty_string_list, struct_to_json_value,
};
use openshell_core::{
AppArmorProfile, Error, ImagePullPolicy, Result as CoreResult, UpstreamProxyConfig,
AppArmorProfile, Error, ImagePullPolicy, NetworkSupervisorTrustBundle, Result as CoreResult,
UpstreamProxyConfig,
};
use openshell_isolation_interface::contract::ResolvedWorkloadIdentity;
use openshell_sandbox_backend::boundary_protocol::{
@@ -138,6 +139,9 @@ const SUPERVISOR_AUTH_BUNDLE_FILE: &str = "supervisor-auth.json";
const START_GENERATION_FILE: &str = "start-generation";
const HOST_OPEN_SHELL_INTERNAL: &str = "host.openshell.internal";
const HOST_DOCKER_INTERNAL: &str = "host.docker.internal";
/// Fixed in-container path for the gateway-owned additional destination trust.
/// It is mounted only into the separate trusted supervisor container.
const ADDITIONAL_CA_MOUNT_PATH: &str = openshell_core::driver_utils::ADDITIONAL_CA_MOUNT_PATH;
fn provisioning_span(
parent: &opentelemetry::Context,
@@ -370,6 +374,9 @@ struct DockerDriverRuntimeConfig {
proxy_ca_bundle: Option<PathBuf>,
provider_spiffe_workload_api_socket: Option<PathBuf>,
app_armor_profile: Option<AppArmorProfile>,
/// Immutable gateway-normalized additional destination trust. This is
/// deliberately separate from `DockerComputeConfig` and sandbox input.
network_trust: Option<NetworkSupervisorTrustBundle>,
}
#[derive(Debug, Clone, Copy)]
@@ -897,6 +904,7 @@ impl DockerComputeDriver {
gateway_bind_address: SocketAddr,
gateway_log_level: &str,
docker_config: &DockerComputeConfig,
network_trust: Option<NetworkSupervisorTrustBundle>,
) -> CoreResult<Self> {
docker_config.validate_configuration(gateway_bind_address)?;
let socket_path = docker_config
@@ -1032,6 +1040,7 @@ impl DockerComputeDriver {
.provider_spiffe_workload_api_socket
.clone(),
app_armor_profile: docker_config.app_armor_profile.clone(),
network_trust,
},
events: broadcast::channel(WATCH_BUFFER).0,
pending: Arc::new(Mutex::new(HashMap::new())),
@@ -1555,6 +1564,7 @@ impl DockerComputeDriver {
}
async fn create_sandbox_inner(&self, sandbox: &DriverSandbox) -> Result<(), Status> {
self.verify_network_trust()?;
let validated = Self::validated_sandbox(sandbox, &self.config)?;
Self::validate_sandbox_auth(sandbox)?;
self.validate_user_volume_mounts_available(&validated.driver_config, &sandbox.workspace)
@@ -1633,6 +1643,7 @@ impl DockerComputeDriver {
}
}
/// Provision from gateway-owned immutable input.
async fn provision_sandbox_inner(
&self,
sandbox: &DriverSandbox,
@@ -1642,6 +1653,10 @@ impl DockerComputeDriver {
DockerProvisioningFailure::new("ContainerCreateFailed", status.message())
})?;
let template = validated.template;
let cleanup_runtime_volumes = || async {
let _ =
remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config).await;
};
let image = async {
openshell_otel::record_error_result(
self.ensure_image_available(&sandbox.id, &template.image)
@@ -1668,7 +1683,7 @@ impl DockerComputeDriver {
prepare_docker_boundary_state_dir(sandbox, &self.config).map_err(|status| {
DockerProvisioningFailure::new("BoundaryStateCreateFailed", status.message())
})?;
create_docker_channel_volume(&self.docker, sandbox, &self.config)
create_docker_channel_volume(&self.docker, sandbox, &self.config, true)
.await
.map_err(|status| {
cleanup_docker_boundary_state(sandbox, &self.config);
@@ -1677,8 +1692,7 @@ impl DockerComputeDriver {
let token_file_created = match write_sandbox_token_file(sandbox, &self.config).await {
Ok(created) => created,
Err(status) => {
let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config)
.await;
cleanup_runtime_volumes().await;
cleanup_docker_boundary_state(sandbox, &self.config);
return Err(DockerProvisioningFailure::new(
"SandboxTokenWriteFailed",
@@ -1687,8 +1701,7 @@ impl DockerComputeDriver {
}
};
if !token_file_created {
let _ =
remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config).await;
cleanup_runtime_volumes().await;
cleanup_docker_boundary_state(sandbox, &self.config);
return Err(DockerProvisioningFailure::new(
"SandboxTokenWriteFailed",
@@ -1707,8 +1720,7 @@ impl DockerComputeDriver {
{
Ok(devices) => devices,
Err(status) => {
let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config)
.await;
cleanup_runtime_volumes().await;
cleanup_docker_boundary_state(sandbox, &self.config);
return Err(DockerProvisioningFailure::new(
"ContainerCreateFailed",
@@ -1726,8 +1738,7 @@ impl DockerComputeDriver {
) {
Ok(body) => body,
Err(status) => {
let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config)
.await;
cleanup_runtime_volumes().await;
cleanup_docker_boundary_state(sandbox, &self.config);
return Err(DockerProvisioningFailure::new(
"ContainerCreateFailed",
@@ -1781,8 +1792,7 @@ impl DockerComputeDriver {
let created = match create_result {
Ok(created) => created,
Err(error) => {
let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config)
.await;
cleanup_runtime_volumes().await;
cleanup_docker_boundary_state(sandbox, &self.config);
return Err(DockerProvisioningFailure::from_status(
"ContainerCreateFailed",
@@ -1800,8 +1810,7 @@ impl DockerComputeDriver {
Some(RemoveContainerOptionsBuilder::default().force(true).build()),
)
.await;
let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config)
.await;
cleanup_runtime_volumes().await;
cleanup_docker_boundary_state(sandbox, &self.config);
return Err(DockerProvisioningFailure::from_status(
"OuterFenceInspectFailed",
@@ -1822,8 +1831,7 @@ impl DockerComputeDriver {
Some(RemoveContainerOptionsBuilder::default().force(true).build()),
)
.await;
let _ =
remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config).await;
cleanup_runtime_volumes().await;
cleanup_docker_boundary_state(sandbox, &self.config);
return Err(DockerProvisioningFailure::from_status(
"OuterFenceRejected",
@@ -1857,8 +1865,7 @@ impl DockerComputeDriver {
Some(RemoveContainerOptionsBuilder::default().force(true).build()),
)
.await;
let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config)
.await;
cleanup_runtime_volumes().await;
cleanup_docker_boundary_state(sandbox, &self.config);
return Err(DockerProvisioningFailure::new(
"BoundaryConfigWriteFailed",
@@ -1897,8 +1904,7 @@ impl DockerComputeDriver {
);
}
cleanup_docker_boundary_state(sandbox, &self.config);
let _ =
remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config).await;
cleanup_runtime_volumes().await;
return Err(DockerProvisioningFailure::from_status(
"ContainerStartFailed",
create_status_from_docker_error("start docker sandbox container", err),
@@ -1933,8 +1939,7 @@ impl DockerComputeDriver {
Some(RemoveContainerOptionsBuilder::default().force(true).build()),
)
.await;
let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config)
.await;
cleanup_runtime_volumes().await;
cleanup_docker_boundary_state(sandbox, &self.config);
return Err(DockerProvisioningFailure::new(
supervisor_start_failure_reason(&status, "ControlSupervisorStartFailed"),
@@ -1978,6 +1983,20 @@ impl DockerComputeDriver {
self.runtime_failures.lock().await.remove(sandbox_id);
}
/// Re-read the immutable artifact directly before any container launch or
/// stopped-pair reconciliation. The bundle never appears in a sandbox
/// request or workload environment, and errors redact its PEM contents.
fn verify_network_trust(&self) -> Result<(), Status> {
if let Some(bundle) = self.config.network_trust.as_ref() {
bundle.verify_artifact().map_err(|error| {
Status::failed_precondition(format!(
"verify gateway-owned network additional CA artifact: {error}"
))
})?;
}
Ok(())
}
fn control_failure_context(
&self,
sandbox: DriverSandbox,
@@ -2076,6 +2095,9 @@ impl DockerComputeDriver {
}
async fn reconcile_runtime_resources_at_startup(&self) -> Result<(), Status> {
// Startup reconciliation must not inspect a changed artifact and then
// leave a trusted supervisor eligible for launch.
self.verify_network_trust()?;
let sandboxes = self.list_managed_container_summaries().await?;
let sandbox_ids = sandboxes
.iter()
@@ -2449,6 +2471,7 @@ impl DockerComputeDriver {
sandbox_name: &str,
generation_id: &str,
launch_authentication: &[u8],
durable_sandbox: Option<&DriverSandbox>,
) -> Result<bool, Status> {
let span_status = openshell_otel::ErrorStatusGuard::current();
require_sandbox_identifier(sandbox_id, sandbox_name)?;
@@ -2464,6 +2487,7 @@ impl DockerComputeDriver {
sandbox_name,
&generation,
launch_authentication,
durable_sandbox,
))
.await;
self.lifecycle_event_fences.finish_start(sandbox_id);
@@ -2476,7 +2500,9 @@ impl DockerComputeDriver {
sandbox_name: &str,
generation: &openshell_core::sandbox_generation::SandboxGenerationId,
launch_authentication: &[u8],
_durable_sandbox: Option<&DriverSandbox>,
) -> Result<bool, Status> {
self.verify_network_trust()?;
let Some(container) = self
.find_managed_container_summary(sandbox_id, sandbox_name)
.await?
@@ -2509,6 +2535,11 @@ impl DockerComputeDriver {
.as_ref()
.and_then(|labels| labels.get(LABEL_SANDBOX_ID))
.map_or(sandbox_id, String::as_str);
// No destination-trust gating here: the trusted supervisor container
// is re-created from the gateway's current configuration on every
// start (see `ensure_control_process_for_container`), so stop/start
// picks up the current additional CA bundle without ever touching
// the workload container or its data.
if !container_state_needs_start(state) {
adopt_or_verify_docker_start_generation(resolved_sandbox_id, &self.config, generation)
.await?;
@@ -3441,6 +3472,7 @@ impl ComputeDriver for DockerComputeDriver {
&request.name,
&request.generation_id,
&request.launch_authentication,
request.sandbox.as_ref(),
))
.await?
{
@@ -4165,6 +4197,7 @@ async fn create_docker_channel_volume(
docker: &Docker,
sandbox: &DriverSandbox,
config: &DockerDriverRuntimeConfig,
create: bool,
) -> Result<(), Status> {
let names = [
docker_channel_volume_name(sandbox, config),
@@ -4173,7 +4206,7 @@ async fn create_docker_channel_volume(
];
for (index, name) in names.iter().enumerate() {
if let Err(error) =
create_docker_runtime_volume(docker, sandbox, config, name.clone()).await
create_docker_runtime_volume(docker, sandbox, config, name.clone(), create).await
{
for created in &names[..index] {
let _ = remove_docker_volume(docker, created).await;
@@ -4189,6 +4222,7 @@ async fn create_docker_runtime_volume(
sandbox: &DriverSandbox,
config: &DockerDriverRuntimeConfig,
name: String,
create: bool,
) -> Result<(), Status> {
let expected_labels = HashMap::from([
(
@@ -4205,16 +4239,18 @@ async fn create_docker_runtime_volume(
LABEL_ISOLATION_BACKEND_OPEN_SHELL.to_string(),
),
]);
docker
.create_volume(VolumeCreateRequest {
name: Some(name.clone()),
labels: Some(expected_labels.clone()),
..Default::default()
})
.await
.map_err(|error| {
Status::internal(format!("create Docker sandbox runtime volume: {error}"))
})?;
if create {
docker
.create_volume(VolumeCreateRequest {
name: Some(name.clone()),
labels: Some(expected_labels.clone()),
..Default::default()
})
.await
.map_err(|error| {
Status::internal(format!("create Docker sandbox runtime volume: {error}"))
})?;
}
let volume = docker.inspect_volume(&name).await.map_err(|error| {
Status::internal(format!("inspect Docker sandbox channel volume: {error}"))
})?;
@@ -4297,10 +4333,14 @@ async fn write_sandbox_token_file(
let Some(spec) = sandbox.spec.as_ref() else {
return Ok(false);
};
if spec.sandbox_token.is_empty() {
return Ok(false);
}
let path = sandbox_token_host_path(sandbox, config)?;
if spec.sandbox_token.is_empty() {
// Restart reconciliation receives a durable public Sandbox, which
// intentionally excludes the bearer token. Its existing private token
// file remains the source of truth while the stopped workload is held
// as rollback state.
return Ok(path.is_file());
}
if let Some(parent) = path.parent() {
openshell_core::paths::create_dir_restricted(parent).map_err(|err| {
Status::internal(format!(
@@ -4594,6 +4634,14 @@ async fn stage_docker_sandbox_bundle(
Ok(())
}
/// Restore the old stopped workload's authenticated channel from the private
/// rollback state. This is used only after a replacement failure; its input is
/// driver-owned files saved before the mutation, never inspected container
/// configuration.
/// Stream a stopped workload's workspace through one bounded anonymous file.
/// Docker's archive endpoint returns an untrusted tar stream, so reject every
/// entry outside the original workspace and all non-file/non-directory entry
/// types before uploading the same filtered archive to the replacement.
fn decode_docker_launch_authentication(
encoded: &[u8],
) -> Result<openshell_core::jwt::SandboxLaunchAuthentication, Status> {
@@ -5245,12 +5293,58 @@ fn docker_supervisor_host_address(grpc_endpoint: &str) -> Option<IpAddr> {
}
}
fn docker_network_trust_supervisor_args(
bundle: Option<&NetworkSupervisorTrustBundle>,
) -> Vec<String> {
bundle.map_or_else(Vec::new, |_bundle| {
vec![
"--additional-ca-bundle".to_string(),
ADDITIONAL_CA_MOUNT_PATH.to_string(),
]
})
}
fn docker_network_trust_supervisor_mount(
bundle: &NetworkSupervisorTrustBundle,
) -> Result<Mount, Status> {
bundle.verify_artifact().map_err(|error| {
Status::failed_precondition(format!(
"verify gateway-owned network additional CA artifact: {error}"
))
})?;
let source = bundle.artifact_path().to_str().ok_or_else(|| {
Status::failed_precondition(format!(
"network additional CA artifact path is not valid UTF-8: {}",
bundle.artifact_path().display()
))
})?;
driver_mounts::validate_absolute_mount_source(source, "network additional CA artifact")
.map_err(Status::failed_precondition)?;
Ok(Mount {
target: Some(ADDITIONAL_CA_MOUNT_PATH.to_string()),
source: Some(source.to_string()),
typ: Some(MountTypeEnum::BIND),
read_only: Some(true),
..Default::default()
})
}
async fn spawn_docker_control_process(
docker: &Docker,
sandbox: &DriverSandbox,
config: &DockerDriverRuntimeConfig,
failure_context: DockerRuntimeFailureContext,
) -> Result<DockerControlProcess, Status> {
// This is the last gateway-side check before the trusted container gets a
// bind mount. It catches replacement/tamper after workload preparation
// without ever exposing the artifact to that workload.
if let Some(bundle) = config.network_trust.as_ref() {
bundle.verify_artifact().map_err(|error| {
Status::failed_precondition(format!(
"verify gateway-owned network additional CA artifact: {error}"
))
})?;
}
let (network_mode, supervisor_endpoint) = docker_supervisor_networking(config)?;
info!(sandbox_id = %sandbox.id, ?network_mode, endpoint = %supervisor_endpoint,
"Selected Docker supervisor networking");
@@ -5356,6 +5450,9 @@ async fn spawn_docker_control_process(
));
let runtime_initiated = read_docker_runtime_descriptor(&sandbox.id, config).await?
.is_some_and(|descriptor| matches!(descriptor.transport, openshell_sandbox_backend::boundary_protocol::SandboxTransport::RuntimeInitiatedUnix { .. }));
command.extend(docker_network_trust_supervisor_args(
config.network_trust.as_ref(),
));
let mut supervisor_mounts = vec![
Mount {
target: Some(BOUNDARY_MOUNT_PATH.to_string()),
@@ -5394,6 +5491,9 @@ async fn spawn_docker_control_process(
});
}
if let Some(bundle) = config.network_trust.as_ref() {
supervisor_mounts.push(docker_network_trust_supervisor_mount(bundle)?);
}
if let Some(socket) = config.provider_spiffe_workload_api_socket.as_ref() {
let parent = socket.parent().ok_or_else(|| {
Status::failed_precondition("provider SPIFFE socket has no parent directory")
@@ -5795,6 +5895,9 @@ async fn stop_docker_control_process(mut process: DockerControlProcess) {
let _ = process.task.await;
}
/// Make a private, on-disk rollback copy of the driver state before a stopped
/// workload is replaced. Only regular files are copied: a compromised state
/// directory cannot smuggle symlinks into the gateway filesystem.
fn cleanup_docker_boundary_state(sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig) {
cleanup_docker_boundary_state_by_id(&sandbox.id, config);
}
crates/openshell-driver-docker/src/main.rs
+1 -1
View File
@@ -97,7 +97,7 @@ async fn main() -> Result<()> {
schema_version: args.otlp_ocsf_schema_version.clone(),
}
});
let driver = DockerComputeDriver::new(args.gateway_bind, &args.log_level, &docker_config)
let driver = DockerComputeDriver::new(args.gateway_bind, &args.log_level, &docker_config, None)
.await
.into_diagnostic()?;
crates/openshell-driver-docker/src/tests.rs
+136 -2
View File
@@ -345,6 +345,7 @@ fn runtime_config() -> DockerDriverRuntimeConfig {
proxy_ca_bundle: None,
provider_spiffe_workload_api_socket: None,
app_armor_profile: Some(AppArmorProfile::Unconfined),
network_trust: None,
}
}
@@ -428,7 +429,7 @@ async fn docker_constructor_rejects_proxy_ca_bundle_without_proxy() {
config.upstream_proxy.https_proxy = None;
let Err(error) =
DockerComputeDriver::new("127.0.0.1:17670".parse().unwrap(), "info", &config).await
DockerComputeDriver::new("127.0.0.1:17670".parse().unwrap(), "info", &config, None).await
else {
panic!("constructor must reject incoherent proxy CA configuration before Docker I/O");
};
@@ -1037,6 +1038,7 @@ async fn start_sandbox_span_does_not_capture_launch_authentication() {
"sandbox",
"invalid-generation",
b"secret-launch-authentication",
None,
)
.with_subscriber(subscriber)
.await
@@ -1126,7 +1128,8 @@ async fn tracing_direct_start_exports_a_docker_start_span() {
let driver = test_driver_with_config(runtime_config());
Box::pin(
DockerComputeDriver::start_sandbox(&driver, "", "", "", &[]).with_subscriber(subscriber),
DockerComputeDriver::start_sandbox(&driver, "", "", "", &[], None)
.with_subscriber(subscriber),
)
.await
.expect_err("missing identifier should fail");
@@ -2579,6 +2582,15 @@ fn build_container_create_body_replaces_inherited_cmd_with_sandbox_bootstrap() {
BOUNDARY_CONFIG_MOUNT_PATH.to_string(),
])
);
assert!(
create_body
.host_config
.as_ref()
.and_then(|config| config.binds.as_ref())
.is_some_and(|binds| binds
.iter()
.all(|bind| !bind.contains(ADDITIONAL_CA_MOUNT_PATH)))
);
assert_eq!(
create_body
.labels
@@ -2603,6 +2615,128 @@ fn build_container_create_body_replaces_inherited_cmd_with_sandbox_bootstrap() {
assert!(create_body.networking_config.is_none());
}
#[test]
fn additional_destination_trust_is_supervisor_only_and_digest_paired() {
let dir = TempDir::new().unwrap();
let artifact = dir.path().join("normalized-additional-ca.crt");
let pem = b"normalized certificate fixture";
fs::write(&artifact, pem).unwrap();
#[cfg(unix)]
fs::set_permissions(
&artifact,
std::os::unix::fs::PermissionsExt::from_mode(0o444),
)
.unwrap();
let bundle = NetworkSupervisorTrustBundle::new(
pem.to_vec(),
1,
"sha256:paired-digest",
artifact.clone(),
);
let mut config = runtime_config();
config.network_trust = Some(bundle.clone());
let workload = build_container_create_body(&test_sandbox(), &config).unwrap();
assert!(
workload
.labels
.as_ref()
.and_then(|labels| labels.get(LABEL_SANDBOX_ID))
.is_some()
);
assert!(
workload
.cmd
.as_ref()
.unwrap()
.iter()
.all(|argument| argument != "--additional-ca-bundle")
);
assert!(
workload
.host_config
.as_ref()
.and_then(|host| host.mounts.as_ref())
.unwrap()
.iter()
.all(|mount| mount.target.as_deref() != Some(ADDITIONAL_CA_MOUNT_PATH))
);
let args = docker_network_trust_supervisor_args(Some(&bundle));
assert!(
args.windows(2)
.any(|arguments| { arguments == ["--additional-ca-bundle", ADDITIONAL_CA_MOUNT_PATH] })
);
let mount = docker_network_trust_supervisor_mount(&bundle).unwrap();
assert_eq!(mount.target.as_deref(), Some(ADDITIONAL_CA_MOUNT_PATH));
assert_eq!(mount.source.as_deref(), artifact.to_str());
assert_eq!(mount.read_only, Some(true));
}
#[test]
fn tampered_additional_destination_trust_fails_closed_without_pem_contents() {
let dir = TempDir::new().unwrap();
let artifact = dir.path().join("normalized-additional-ca.crt");
fs::write(&artifact, b"expected").unwrap();
#[cfg(unix)]
fs::set_permissions(
&artifact,
std::os::unix::fs::PermissionsExt::from_mode(0o444),
)
.unwrap();
let bundle = NetworkSupervisorTrustBundle::new(
b"expected".to_vec(),
1,
"sha256:expected",
artifact.clone(),
);
fs::remove_file(&artifact).unwrap();
fs::write(&artifact, b"BEGIN CERTIFICATE modified secret data").unwrap();
#[cfg(unix)]
fs::set_permissions(
&artifact,
std::os::unix::fs::PermissionsExt::from_mode(0o444),
)
.unwrap();
let error = docker_network_trust_supervisor_mount(&bundle)
.expect_err("a changed gateway artifact must fail closed");
assert!(error.message().contains("network additional CA artifact"));
assert!(!error.message().contains("modified secret data"));
}
#[test]
fn user_mount_cannot_replace_network_trust_destination() {
let dir = TempDir::new().unwrap();
let source = dir.path().join("user-ca.crt");
fs::write(&source, b"user-controlled fixture").unwrap();
let mut config = runtime_config();
config.enable_bind_mounts = true;
let mut sandbox = test_sandbox();
sandbox
.spec
.as_mut()
.unwrap()
.template
.as_mut()
.unwrap()
.driver_config = Some(json_struct(serde_json::json!({
"mounts": [{
"type": "bind",
"source": source,
"target": ADDITIONAL_CA_MOUNT_PATH
}]
})));
let error = build_container_create_body(&sandbox, &config)
.expect_err("user mount must not mask network trust material");
assert_eq!(error.code(), tonic::Code::FailedPrecondition);
assert!(
error.message().contains("reserved OpenShell path"),
"unexpected error: {error}"
);
}
#[test]
fn validate_sandbox_rejects_gpu_when_cdi_unavailable() {
let config = runtime_config();
crates/openshell-driver-kubernetes/Cargo.toml
+1
View File
@@ -36,6 +36,7 @@ k8s-openapi = { workspace = true }
tower = { workspace = true, features = ["timeout"] }
serde = { workspace = true }
serde_json = { workspace = true }
sha2 = { workspace = true }
clap = { workspace = true }
tracing = { workspace = true }
tracing-subscriber = { workspace = true }
crates/openshell-driver-kubernetes/src/driver.rs
+455 -4
View File
@@ -31,8 +31,8 @@ use k8s_openapi::api::authentication::v1::{
TokenReview, TokenReviewSpec, TokenReviewStatus, UserInfo,
};
use k8s_openapi::api::core::v1::{
Event as KubeEventObj, Namespace, Node, PersistentVolumeClaimVolumeSource, Pod, Secret,
Service, ServiceAccount, Volume, VolumeMount,
ConfigMap, Event as KubeEventObj, Namespace, Node, PersistentVolumeClaimVolumeSource, Pod,
Secret, Service, ServiceAccount, Volume, VolumeMount,
};
use k8s_openapi::api::networking::v1::{
NetworkPolicy, NetworkPolicyIngressRule, NetworkPolicyPeer, NetworkPolicyPort,
@@ -49,6 +49,7 @@ use kube::runtime::WatchStreamExt;
use kube::runtime::wait::await_condition;
use kube::runtime::watcher::{self, Event};
use kube::{Client, Error as KubeError};
use openshell_core::NetworkSupervisorTrustBundle;
use openshell_core::driver_mounts;
use openshell_core::driver_utils::{
LABEL_GATEWAY_ID, LABEL_MANAGED_BY, LABEL_MANAGED_BY_VALUE, LABEL_SANDBOX_ID,
@@ -79,6 +80,7 @@ use openshell_supervisor_control::{
RegistrationTraceContext, SandboxInstanceRegistration, SharedSupervision,
};
use serde::Deserialize;
use sha2::{Digest as _, Sha256};
use std::collections::{BTreeMap, HashSet};
use std::path::{Path, PathBuf};
use std::pin::Pin;
@@ -125,6 +127,194 @@ const ANNOTATION_SANDBOX_RUNTIME_CONTROL_POLICY_UID: &str =
const ANNOTATION_SANDBOX_RUNTIME_CONTROL_POLICY_GENERATION: &str =
"openshell.ai/sandbox-runtime-control-policy-generation";
/// Full gateway-normalized destination-trust digest recorded on each immutable
/// `ConfigMap` generation. The digest is intentionally not truncated here even
/// though the object name uses stable, DNS-safe hash prefixes.
const ANNOTATION_ADDITIONAL_CA_GENERATION: &str = "openshell.ai/additional-ca-generation";
const ADDITIONAL_CA_CONFIG_MAP_PREFIX: &str = "openshell-additional-ca-";
const ADDITIONAL_CA_DATA_KEY: &str = "ca.crt";
/// A validated immutable `ConfigMap` generation selected for a supervisor Pod.
/// This contains only an object name and non-secret digest; PEM bytes remain in
/// the API object and are never copied into a workload template or Secret.
#[derive(Clone, Debug, PartialEq, Eq)]
struct NetworkAdditionalCaConfigMap {
name: String,
digest: String,
}
fn network_additional_ca_config_map_name(gateway_id: &str, digest: &str) -> String {
// Kubernetes labels have a 63-byte limit. Keep recognizable stable
// prefixes while hashing both variable inputs. The complete digest is
// retained in an annotation and checked before every supervisor creation.
// The hashes derive deterministic object names, not authenticity proofs;
// trust comes from the immutable flag, the digest annotation, and the
// exact-byte validation. Classified as a non-security identity hash in
// openshell-crypto/README.md.
let gateway_hash = format!("{:x}", Sha256::digest(gateway_id.as_bytes()));
let digest_hash = format!("{:x}", Sha256::digest(digest.as_bytes()));
format!(
"{ADDITIONAL_CA_CONFIG_MAP_PREFIX}{}-{}",
&gateway_hash[..12],
&digest_hash[..26],
)
}
fn network_additional_ca_config_map(
namespace: &str,
gateway_id: &str,
bundle: &NetworkSupervisorTrustBundle,
) -> Result<ConfigMap, KubernetesDriverError> {
let pem = std::str::from_utf8(bundle.normalized_pem()).map_err(|_| {
KubernetesDriverError::Precondition(
"gateway-normalized network additional CA bundle is not UTF-8".to_string(),
)
})?;
let name = network_additional_ca_config_map_name(gateway_id, bundle.digest());
Ok(ConfigMap {
metadata: ObjectMeta {
name: Some(name),
namespace: Some(namespace.to_string()),
labels: Some(BTreeMap::from([
(
LABEL_MANAGED_BY.to_string(),
LABEL_MANAGED_BY_VALUE.to_string(),
),
(LABEL_GATEWAY_ID.to_string(), gateway_id.to_string()),
])),
annotations: Some(BTreeMap::from([(
ANNOTATION_ADDITIONAL_CA_GENERATION.to_string(),
bundle.digest().to_string(),
)])),
// Generations are intentionally retained across Pod and Sandbox
// replacement. A sandbox owner reference would allow garbage
// collection to remove material still needed by a stopped PVC.
owner_references: None,
..Default::default()
},
immutable: Some(true),
data: Some(BTreeMap::from([(
ADDITIONAL_CA_DATA_KEY.to_string(),
pem.to_string(),
)])),
binary_data: None,
})
}
fn validate_network_additional_ca_config_map(
existing: &ConfigMap,
expected: &ConfigMap,
) -> Result<(), KubernetesDriverError> {
let name = expected.metadata.name.as_deref().unwrap_or_default();
// Validate the gateway-owned label pairs individually. A mutating
// admission webhook may attach organizational labels to the created
// ConfigMap; an exact-map comparison would then reject the correct map on
// the create response and on every later create/start through this
// existing-object path, blocking sandbox launches despite the required
// ownership labels, immutable flag, digest, and certificate bytes
// remaining correct. Unrelated labels are therefore allowed; each
// gateway-owned pair must match exactly.
if let Some(expected_labels) = expected.metadata.labels.as_ref() {
let existing_labels = existing.metadata.labels.as_ref().ok_or_else(|| {
KubernetesDriverError::Precondition(format!(
"network additional CA ConfigMap {name} is not owned by this gateway"
))
})?;
for (key, value) in expected_labels {
if existing_labels.get(key) != Some(value) {
return Err(KubernetesDriverError::Precondition(format!(
"network additional CA ConfigMap {name} is not owned by this gateway"
)));
}
}
}
// The immutable generation must outlive individual sandbox objects.
// Reject even an explicitly empty ownerReferences field so only the
// API representation produced by our ownerless template is accepted.
if existing.metadata.owner_references.is_some() {
return Err(KubernetesDriverError::Precondition(format!(
"network additional CA ConfigMap {name} is not owned by this gateway"
)));
}
let expected_digest = expected
.metadata
.annotations
.as_ref()
.and_then(|annotations| annotations.get(ANNOTATION_ADDITIONAL_CA_GENERATION));
let actual_digest = existing
.metadata
.annotations
.as_ref()
.and_then(|annotations| annotations.get(ANNOTATION_ADDITIONAL_CA_GENERATION));
if actual_digest != expected_digest {
return Err(KubernetesDriverError::Precondition(format!(
"network additional CA ConfigMap {name} has an unexpected trust generation"
)));
}
if existing.immutable != Some(true) {
return Err(KubernetesDriverError::Precondition(format!(
"network additional CA ConfigMap {name} is not immutable"
)));
}
if existing.data != expected.data || existing.binary_data.is_some() {
return Err(KubernetesDriverError::Precondition(format!(
"network additional CA ConfigMap {name} did not retain the exact gateway-normalized trust bundle"
)));
}
Ok(())
}
async fn create_or_validate_network_additional_ca_config_map(
config_maps: &Api<ConfigMap>,
expected: &ConfigMap,
) -> Result<(), KubernetesDriverError> {
let name = expected.metadata.name.as_deref().ok_or_else(|| {
KubernetesDriverError::Message("network additional CA ConfigMap has no name".to_string())
})?;
match tokio::time::timeout(KUBE_API_TIMEOUT, config_maps.get_opt(name)).await {
Ok(Ok(Some(existing))) => {
return validate_network_additional_ca_config_map(&existing, expected);
}
Ok(Ok(None)) => {}
Ok(Err(error)) => return Err(KubernetesDriverError::from_kube(error)),
Err(_) => {
return Err(KubernetesDriverError::Message(
"timed out reading network additional CA ConfigMap".to_string(),
));
}
}
match tokio::time::timeout(
KUBE_API_TIMEOUT,
config_maps.create(&PostParams::default(), expected),
)
.await
{
// Validate the server response as well: an admission controller must
// not be able to silently alter the immutable material we mount.
Ok(Ok(created)) => validate_network_additional_ca_config_map(&created, expected),
// A concurrent gateway instance may win creation. Re-read exactly the
// named object and subject the winner to the same byte-for-byte checks;
// no LIST, WATCH, PATCH, UPDATE, or DELETE is ever used for this API.
Ok(Err(KubeError::Api(error))) if error.code == 409 => {
let existing = tokio::time::timeout(KUBE_API_TIMEOUT, config_maps.get(name))
.await
.map_err(|_| {
KubernetesDriverError::Message(
"timed out validating concurrent network additional CA ConfigMap creation"
.to_string(),
)
})?
.map_err(KubernetesDriverError::from_kube)?;
validate_network_additional_ca_config_map(&existing, expected)
}
Ok(Err(error)) => Err(KubernetesDriverError::from_kube(error)),
Err(_) => Err(KubernetesDriverError::Message(
"timed out creating network additional CA ConfigMap".to_string(),
)),
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum SandboxRuntimeBootstrapPhase {
Preparing,
@@ -894,6 +1084,10 @@ pub struct KubernetesComputeDriver {
sandbox_api_version: Arc<OnceCell<&'static str>>,
lifecycle_gates: Arc<LifecycleGates>,
config: KubernetesComputeConfig,
/// Gateway-normalized destination trust available only to this in-process
/// driver. The external driver protocol deliberately has no transport for
/// the PEM bytes or a gateway-local artifact path.
network_trust_bundle: Option<NetworkSupervisorTrustBundle>,
operator_allowlist: Option<OperatorNamespaceAllowlist>,
/// Desired shared sandbox instances while new generations use the gateway
/// topology.
@@ -945,6 +1139,7 @@ impl KubernetesComputeDriver {
sandbox_api_version: Arc::new(OnceCell::new()),
lifecycle_gates: Arc::default(),
config,
network_trust_bundle: None,
operator_allowlist: None,
shared_supervisor: None,
}
@@ -957,6 +1152,7 @@ impl KubernetesComputeDriver {
pub async fn new(
config: KubernetesComputeConfig,
shutdown_rx: tokio::sync::watch::Receiver<bool>,
network_trust_bundle: Option<NetworkSupervisorTrustBundle>,
) -> Result<Self, KubernetesDriverError> {
if config.shared_supervisor.is_some() {
return Err(KubernetesDriverError::Precondition(
@@ -964,7 +1160,7 @@ impl KubernetesComputeDriver {
.to_string(),
));
}
Self::build(config, shutdown_rx, None).await
Self::build(config, shutdown_rx, None, network_trust_bundle).await
}
/// A driver whose new generations become sandbox instances on the shared
@@ -984,13 +1180,17 @@ impl KubernetesComputeDriver {
.to_string(),
));
}
Self::build(config, shutdown_rx, Some(supervision)).await
// The gateway refuses the additional destination CA under the shared
// supervisor topology fail-closed, so no trust bundle reaches this
// constructor.
Self::build(config, shutdown_rx, Some(supervision), None).await
}
async fn build(
config: KubernetesComputeConfig,
shutdown_rx: tokio::sync::watch::Receiver<bool>,
supervision: Option<SharedSupervision>,
network_trust_bundle: Option<NetworkSupervisorTrustBundle>,
) -> Result<Self, KubernetesDriverError> {
config
.resource_admission
@@ -1090,6 +1290,7 @@ impl KubernetesComputeDriver {
sandbox_api_version: Arc::new(OnceCell::new()),
lifecycle_gates: Arc::default(),
config,
network_trust_bundle,
operator_allowlist,
shared_supervisor,
};
@@ -2652,6 +2853,29 @@ impl KubernetesComputeDriver {
Ok(())
}
/// Create or validate the gateway-owned destination-trust generation for
/// one target namespace. The source PEM never crosses the external driver
/// boundary: only this in-process driver receives the normalized bundle.
async fn ensure_network_additional_ca_config_map(
&self,
namespace: &str,
) -> Result<Option<NetworkAdditionalCaConfigMap>, KubernetesDriverError> {
let Some(bundle) = self.network_trust_bundle.as_ref() else {
return Ok(None);
};
let expected =
network_additional_ca_config_map(namespace, &self.config.gateway_id, bundle)?;
let name = expected.metadata.name.clone().ok_or_else(|| {
KubernetesDriverError::Message(
"network additional CA ConfigMap has no name".to_string(),
)
})?;
let digest = bundle.digest().to_string();
let config_maps = Api::<ConfigMap>::namespaced(self.client.clone(), namespace);
create_or_validate_network_additional_ca_config_map(&config_maps, &expected).await?;
Ok(Some(NetworkAdditionalCaConfigMap { name, digest }))
}
async fn wait_for_bootstrap_workload_pod(
&self,
pods: &Api<Pod>,
@@ -3121,6 +3345,12 @@ impl KubernetesComputeDriver {
log_level: &str,
upstream_proxy_ca_bundle_staged: bool,
) -> Result<String, KubernetesDriverError> {
// Validate or create the immutable trust generation before creating
// the dedicated supervisor Pod. A same-name delete/recreate with
// altered bytes therefore fails closed while no control process runs.
let network_additional_ca = self
.ensure_network_additional_ca_config_map(namespace)
.await?;
let pods: Api<Pod> = Api::namespaced(self.client.clone(), namespace);
pods.create(
&PostParams::default(),
@@ -3155,6 +3385,9 @@ impl KubernetesComputeDriver {
.provider_spiffe_workload_api_socket_path
.as_str(),
),
network_additional_ca
.as_ref()
.map(|generation| (generation.name.as_str(), generation.digest.as_str())),
owner,
&self.supervisor_otlp_environment(),
)
@@ -3574,6 +3807,12 @@ impl KubernetesComputeDriver {
vec![workload_owner, supervisor_owner],
)
.await?;
// A restarted generation must revalidate immediately before
// resuming the gated supervisor as well as before creating
// it. The shared topology never carries an additional CA:
// the gateway refuses that combination fail-closed.
self.ensure_network_additional_ca_config_map(namespace)
.await?;
pods.patch(
&names.supervisor_pod,
&PatchParams::default(),
@@ -3930,6 +4169,12 @@ impl KubernetesComputeDriver {
let names = SandboxRuntimeNames::for_generation(sandbox_id, generation.as_str());
self.create_sandbox_runtime_fence(&namespace, &names)
.await?;
// Validate the current gateway generation before touching a stopped
// sandbox's stale companion. This detects equal-generation
// delete/recreate tampering before a supervisor can be created or
// released, while running supervisors remain untouched.
self.ensure_network_additional_ca_config_map(&namespace)
.await?;
let mut stale_generations = sandbox_runtime_generation(&object).as_slice().to_vec();
stale_generations.push(generation.as_str());
self.delete_sandbox_runtime_supervisor(sandbox_id, &namespace, &stale_generations)
@@ -8964,6 +9209,7 @@ mod tests {
sandbox_api_version: Arc::new(OnceCell::new()),
lifecycle_gates: Arc::default(),
config: KubernetesComputeConfig::default(),
network_trust_bundle: None,
operator_allowlist: None,
shared_supervisor: None,
};
@@ -8977,6 +9223,7 @@ mod tests {
assert!(steps.lock().unwrap().is_empty());
}
use crate::sandbox_runtime::{ADDITIONAL_CA_MOUNT_PATH, ADDITIONAL_CA_VOLUME_NAME};
use openshell_core::progress::{
PROGRESS_ACTIVE_DETAIL_KEY, PROGRESS_ACTIVE_STEP_KEY, PROGRESS_COMPLETE_LABEL_KEY,
PROGRESS_COMPLETE_STEP_KEY,
@@ -9092,6 +9339,179 @@ mod tests {
)
}
fn network_trust_bundle(gateway_id: &str, digest: &str) -> NetworkSupervisorTrustBundle {
NetworkSupervisorTrustBundle::new(
b"-----BEGIN CERTIFICATE-----\nfixture\n-----END CERTIFICATE-----\n".to_vec(),
1,
digest,
PathBuf::from(format!("/gateway-state/{gateway_id}.crt")),
)
}
#[test]
fn destination_trust_config_map_is_immutable_content_addressed_and_retained() {
let digest = "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
let bundle = network_trust_bundle("gateway-a", digest);
let expected = network_additional_ca_config_map("workspace-a", "gateway-a", &bundle)
.expect("normalized PEM must be valid UTF-8");
let name = expected.metadata.name.as_deref().expect("ConfigMap name");
assert_eq!(name.len(), 63);
assert!(name.starts_with(ADDITIONAL_CA_CONFIG_MAP_PREFIX));
assert!(
name.bytes()
.all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'-')
);
assert_eq!(expected.immutable, Some(true));
assert_eq!(expected.metadata.owner_references, None);
assert_eq!(
expected.metadata.labels.as_ref(),
Some(&BTreeMap::from([
(
LABEL_MANAGED_BY.to_string(),
LABEL_MANAGED_BY_VALUE.to_string()
),
(LABEL_GATEWAY_ID.to_string(), "gateway-a".to_string()),
]))
);
assert_eq!(
expected
.metadata
.annotations
.as_ref()
.and_then(|annotations| annotations.get(ANNOTATION_ADDITIONAL_CA_GENERATION)),
Some(&digest.to_string())
);
assert_eq!(
expected.data.as_ref(),
Some(&BTreeMap::from([(
ADDITIONAL_CA_DATA_KEY.to_string(),
String::from_utf8(bundle.normalized_pem().to_vec()).unwrap(),
)]))
);
assert_eq!(expected.binary_data, None);
assert_ne!(
name,
network_additional_ca_config_map_name("gateway-b", digest),
"gateway identity participates in content addressing"
);
assert_ne!(
name,
network_additional_ca_config_map_name("gateway-a", "sha256:other-generation"),
"trust generation participates in content addressing"
);
assert!(validate_network_additional_ca_config_map(&expected, &expected).is_ok());
}
#[test]
fn destination_trust_config_map_rejects_same_name_recreation_tampering() {
let bundle = network_trust_bundle(
"gateway-a",
"sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
);
let expected = network_additional_ca_config_map("workspace-a", "gateway-a", &bundle)
.expect("normalized PEM must be valid UTF-8");
let mut replaced = expected.clone();
replaced.data = Some(BTreeMap::from([(
ADDITIONAL_CA_DATA_KEY.to_string(),
"not-a-certificate\n".to_string(),
)]));
assert!(matches!(
validate_network_additional_ca_config_map(&replaced, &expected),
Err(KubernetesDriverError::Precondition(message))
if message.contains("did not retain the exact gateway-normalized trust bundle")
));
let mut binary_data = expected.clone();
binary_data.binary_data = Some(BTreeMap::new());
assert!(matches!(
validate_network_additional_ca_config_map(&binary_data, &expected),
Err(KubernetesDriverError::Precondition(message))
if message.contains("did not retain the exact gateway-normalized trust bundle")
));
let mut wrong_gateway = expected.clone();
wrong_gateway
.metadata
.labels
.as_mut()
.unwrap()
.insert(LABEL_GATEWAY_ID.to_string(), "other-gateway".to_string());
assert!(matches!(
validate_network_additional_ca_config_map(&wrong_gateway, &expected),
Err(KubernetesDriverError::Precondition(message))
if message.contains("is not owned by this gateway")
));
let mut sandbox_owned = expected.clone();
sandbox_owned.metadata.owner_references =
Some(vec![sandbox_runtime_sandbox_owner_reference(
"sandbox",
"sandbox-uid",
"agents.x-k8s.io/v1beta1",
false,
)]);
assert!(matches!(
validate_network_additional_ca_config_map(&sandbox_owned, &expected),
Err(KubernetesDriverError::Precondition(message))
if message.contains("is not owned by this gateway")
));
let mut stale_digest = expected.clone();
stale_digest.metadata.annotations.as_mut().unwrap().insert(
ANNOTATION_ADDITIONAL_CA_GENERATION.to_string(),
"sha256:stale".to_string(),
);
assert!(matches!(
validate_network_additional_ca_config_map(&stale_digest, &expected),
Err(KubernetesDriverError::Precondition(message))
if message.contains("unexpected trust generation")
));
}
#[test]
fn destination_trust_config_map_allows_admission_added_labels() {
// A mutating admission webhook may attach organizational labels to
// the created ConfigMap. The gateway-owned pairs must still match
// exactly, but unrelated labels must not block every later
// create/start through the existing-object validation path.
let bundle = network_trust_bundle(
"gateway-a",
"sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
);
let expected = network_additional_ca_config_map("workspace-a", "gateway-a", &bundle)
.expect("normalized PEM must be valid UTF-8");
let mut admitted = expected.clone();
admitted
.metadata
.labels
.as_mut()
.expect("expected ConfigMap has gateway-owned labels")
.insert(
"org.example/compliance-scan".to_string(),
"managed".to_string(),
);
assert!(
validate_network_additional_ca_config_map(&admitted, &expected).is_ok(),
"an admission-added organizational label must not reject a correct \
ConfigMap"
);
let mut missing_gateway_label = expected.clone();
missing_gateway_label
.metadata
.labels
.as_mut()
.unwrap()
.remove(&LABEL_GATEWAY_ID.to_string());
assert!(matches!(
validate_network_additional_ca_config_map(&missing_gateway_label, &expected),
Err(KubernetesDriverError::Precondition(message))
if message.contains("is not owned by this gateway")
));
}
#[test]
fn boundary_authority_uses_stable_service_dns_name() {
let names = SandboxRuntimeNames::new("sandbox-1");
@@ -10048,6 +10468,7 @@ mod tests {
config: KubernetesComputeConfig::default(),
operator_allowlist: None,
shared_supervisor: None,
network_trust_bundle: None,
};
driver
.sandbox_api_version
@@ -10140,6 +10561,7 @@ mod tests {
config: KubernetesComputeConfig::default(),
operator_allowlist: None,
shared_supervisor: None,
network_trust_bundle: None,
};
driver
@@ -11421,6 +11843,33 @@ mod tests {
);
let volumes = pod_template["spec"]["volumes"].as_array().unwrap();
assert!(
!volumes
.iter()
.any(|volume| volume["name"] == ADDITIONAL_CA_VOLUME_NAME),
"destination trust is exclusive to the separate supervisor Pod"
);
for container in pod_template["spec"]["containers"]
.as_array()
.into_iter()
.flatten()
.chain(
pod_template["spec"]["initContainers"]
.as_array()
.into_iter()
.flatten(),
)
{
assert!(
!container["volumeMounts"].as_array().is_some_and(|mounts| {
mounts.iter().any(|mount| {
mount["name"] == ADDITIONAL_CA_VOLUME_NAME
|| mount["mountPath"] == ADDITIONAL_CA_MOUNT_PATH
})
}),
"destination trust leaked into a workload or init container: {container:#?}"
);
}
let pod_identity = volumes
.iter()
.find(|volume| volume["name"] == SANDBOX_POD_IDENTITY_VOLUME_NAME)
@@ -12878,6 +13327,7 @@ mod tests {
sandbox_api_version: Arc::new(OnceCell::new()),
lifecycle_gates: Arc::default(),
config,
network_trust_bundle: None,
operator_allowlist: None,
shared_supervisor: None,
};
@@ -13183,6 +13633,7 @@ mod tests {
sandbox_api_version: Arc::new(OnceCell::new()),
lifecycle_gates: Arc::default(),
config: managed_config(&[]),
network_trust_bundle: None,
operator_allowlist: None,
shared_supervisor: None,
};
crates/openshell-driver-kubernetes/src/lifecycle_tests.rs
+1
View File
@@ -35,6 +35,7 @@ mod lifecycle_reconciliation {
sandbox_api_version: Arc::new(OnceCell::new()),
lifecycle_gates: Arc::default(),
config: KubernetesComputeConfig::default(),
network_trust_bundle: None,
operator_allowlist: None,
shared_supervisor: None,
};
crates/openshell-driver-kubernetes/src/main.rs
+1
View File
@@ -329,6 +329,7 @@ async fn main() -> Result<()> {
sandbox_gid: args.sandbox_gid,
},
shutdown_rx,
None,
)
.await
.into_diagnostic()?;
crates/openshell-driver-kubernetes/src/sandbox_runtime.rs
+131 -2
View File
@@ -8,8 +8,8 @@ use std::path::Path;
use k8s_openapi::ByteString;
use k8s_openapi::api::core::v1::{
CSIVolumeSource, Capabilities, Container, EmptyDirVolumeSource, EnvVar, KeyToPath,
LocalObjectReference, Pod, PodSchedulingGate, PodSecurityContext, PodSpec, Probe,
CSIVolumeSource, Capabilities, ConfigMapVolumeSource, Container, EmptyDirVolumeSource, EnvVar,
KeyToPath, LocalObjectReference, Pod, PodSchedulingGate, PodSecurityContext, PodSpec, Probe,
ProjectedVolumeSource, Secret, SecretVolumeSource, SecurityContext, Service,
ServiceAccountTokenProjection, ServicePort, ServiceSpec, TCPSocketAction, Volume, VolumeMount,
VolumeProjection,
@@ -67,6 +67,12 @@ pub const CONTROL_HEALTH_SOCKET_PATH: &str = "/run/openshell/health.sock";
/// Kubelet `tcpSocket` readiness port. An exec probe would start a supervisor
/// process in every sandbox on every period.
pub const CONTROL_HEALTH_PORT: u16 = 5501;
/// `ConfigMap` volume name reserved for gateway-normalized destination trust.
pub const ADDITIONAL_CA_VOLUME_NAME: &str = "openshell-additional-ca";
/// The only destination-trust path exposed to the dedicated supervisor Pod.
pub const ADDITIONAL_CA_MOUNT_PATH: &str = openshell_core::driver_utils::ADDITIONAL_CA_MOUNT_PATH;
const ADDITIONAL_CA_KEY: &str = "ca.crt";
pub const NAMESPACE_WORKLOAD_POLICY_NAME: &str = "openshell-sandbox-workloads";
pub const NAMESPACE_SUPERVISOR_EGRESS_POLICY_NAME: &str = "openshell-sandbox-supervisors";
pub const SUPERVISOR_TERMINATION_GRACE_PERIOD_SECONDS: i64 = 30;
@@ -341,6 +347,10 @@ pub fn supervisor_pod(
proxy_connect_by_hostname: bool,
upstream_proxy_ca_bundle_staged: bool,
provider_spiffe_socket_path: Option<&str>,
// Name and expected full digest of a gateway-normalized destination trust
// generation. The ConfigMap is mounted only in this dedicated control Pod;
// it is deliberately absent from the workload Pod and its bootstrap Secret.
network_additional_ca: Option<(&str, &str)>,
owner: OwnerReference,
tracing_environment: &[(&str, String)],
) -> Result<Pod, String> {
@@ -481,6 +491,22 @@ pub fn supervisor_pod(
UPSTREAM_PROXY_CA_BUNDLE_PATH.to_string(),
]);
}
if let Some((config_map_name, _digest)) = network_additional_ca {
command.extend([
"--additional-ca-bundle".to_string(),
ADDITIONAL_CA_MOUNT_PATH.to_string(),
]);
let mut mount = volume_mount(ADDITIONAL_CA_VOLUME_NAME, ADDITIONAL_CA_MOUNT_PATH, true);
// A subPath exposes `ca.crt` at the exact protected file path rather
// than mounting a ConfigMap directory over the supervisor's filesystem.
mount.sub_path = Some(ADDITIONAL_CA_KEY.to_string());
volume_mounts.push(mount);
volumes.push(config_map_volume(
ADDITIONAL_CA_VOLUME_NAME,
config_map_name,
ADDITIONAL_CA_KEY,
));
}
if let Some((secret_name, secret_key)) = proxy_auth_secret {
let auth_path = Path::new(openshell_core::container_paths::UPSTREAM_PROXY_AUTH_MOUNT_PATH);
let mount_path = auth_path
@@ -828,6 +854,23 @@ fn volume_mount(name: &str, mount_path: &str, read_only: bool) -> VolumeMount {
}
}
fn config_map_volume(name: &str, config_map_name: &str, key: &str) -> Volume {
Volume {
name: name.to_string(),
config_map: Some(ConfigMapVolumeSource {
name: config_map_name.to_string(),
default_mode: Some(0o444),
items: Some(vec![KeyToPath {
key: key.to_string(),
path: key.to_string(),
..Default::default()
}]),
..Default::default()
}),
..Default::default()
}
}
fn secret_volume(name: &str, secret_name: &str, item: Option<KeyToPath>) -> Volume {
Volume {
name: name.to_string(),
@@ -999,6 +1042,7 @@ mod tests {
false,
false,
None,
None,
owner(),
&[],
)
@@ -1058,6 +1102,7 @@ mod tests {
false,
false,
None,
None,
owner(),
&[
(
@@ -1184,6 +1229,7 @@ mod tests {
false,
false,
None,
None,
owner(),
&[],
)
@@ -1323,6 +1369,88 @@ mod tests {
assert_eq!(mount.read_only, Some(true));
}
#[test]
fn destination_trust_is_mounted_only_into_the_dedicated_supervisor_pod() {
let names = SandboxRuntimeNames::new("pair");
let digest = "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
let pod = supervisor_pod(
"sandbox",
&names,
"pair",
"demo",
"gateway",
"supervisor:latest",
None,
"sandbox-sa",
1000,
1000,
&[],
"https://gateway:8080",
SupervisorClientTls::Disabled,
"{}",
"info",
600,
None,
None,
None,
false,
false,
false,
None,
Some(("openshell-additional-ca-gateway-generation", digest)),
owner(),
&[],
)
.expect("render supervisor Pod with destination trust");
let spec = pod.spec.expect("Pod spec");
let container = &spec.containers[0];
assert!(container.command.as_deref().is_some_and(|command| {
command
.windows(2)
.any(|args| args == ["--additional-ca-bundle", ADDITIONAL_CA_MOUNT_PATH])
}));
assert_eq!(
container
.volume_mounts
.as_deref()
.unwrap_or_default()
.iter()
.filter(|mount| mount.name == ADDITIONAL_CA_VOLUME_NAME)
.collect::<Vec<_>>()
.as_slice(),
&[&VolumeMount {
name: ADDITIONAL_CA_VOLUME_NAME.to_string(),
mount_path: ADDITIONAL_CA_MOUNT_PATH.to_string(),
read_only: Some(true),
sub_path: Some(ADDITIONAL_CA_KEY.to_string()),
..Default::default()
}]
);
let volume = spec
.volumes
.as_deref()
.unwrap_or_default()
.iter()
.find(|volume| volume.name == ADDITIONAL_CA_VOLUME_NAME)
.expect("destination trust ConfigMap volume");
assert_eq!(
volume
.config_map
.as_ref()
.map(|source| source.name.as_str()),
Some("openshell-additional-ca-gateway-generation")
);
assert_eq!(
volume
.config_map
.as_ref()
.and_then(|source| source.items.as_deref())
.and_then(|items| items.first())
.map(|item| (item.key.as_str(), item.path.as_str())),
Some((ADDITIONAL_CA_KEY, ADDITIONAL_CA_KEY))
);
}
#[test]
fn bootstrap_secrets_are_immutable_and_split_by_trust_domain() {
let names = SandboxRuntimeNames::new("pair");
@@ -1442,6 +1570,7 @@ mod tests {
false,
staged,
None,
None,
owner(),
&[],
)
crates/openshell-driver-podman/Cargo.toml
+2 -1
View File
@@ -20,6 +20,7 @@ openshell-otel = { path = "../openshell-otel" }
openshell-isolation-interface = { path = "../openshell-isolation-interface" }
openshell-sandbox-backend = { path = "../openshell-sandbox-backend" }
tar = "0.4"
tempfile = "3"
uuid = { workspace = true }
tokio = { workspace = true }
@@ -44,6 +45,7 @@ tower-http = { workspace = true }
http = { workspace = true }
thiserror = { workspace = true }
miette = { workspace = true }
sha2 = { workspace = true }
url = { workspace = true }
[dev-dependencies]
@@ -53,7 +55,6 @@ opentelemetry_sdk = { workspace = true, features = ["testing"] }
prost-types = { workspace = true }
rcgen = { workspace = true }
temp-env = "0.3"
tempfile = "3"
tokio = { workspace = true, features = ["test-util"] }
[lints]
crates/openshell-driver-podman/src/client.rs
+117 -1
View File
@@ -47,7 +47,7 @@ pub enum PodmanApiError {
/// Maximum resource name length. Podman container names become directory
/// names in the storage driver, so we cap at 255 to stay within ext4/xfs
/// filename limits.
const MAX_NAME_LEN: usize = 255;
pub const MAX_NAME_LEN: usize = 255;
/// Validate that a resource name is safe for URL path interpolation.
///
@@ -172,6 +172,12 @@ pub struct ContainerConfig {
pub labels: HashMap<String, String>,
#[serde(default)]
pub user: String,
/// The immutable image ID the container was created from. Podman records
/// the reference used at creation time here; workloads are always created
/// from a resolved image ID, so this pins the image the container runs
/// independently of any later tag movement.
#[serde(default)]
pub image: String,
}
/// Immutable image metadata needed to bind OCI identity inspection to launch.
@@ -662,6 +668,40 @@ impl PodmanClient {
}
}
/// Rename a stopped container by name or ID.
///
/// The libpod rename endpoint frees the old canonical name without
/// deleting the container. Reconciliation uses it as a rollback point
/// while a replacement is created.
pub async fn rename_container(&self, name: &str, new_name: &str) -> Result<(), PodmanApiError> {
validate_name(name)?;
validate_name(new_name)?;
self.request_ok(
hyper::Method::POST,
&format!("/libpod/containers/{name}/rename?name={new_name}"),
None,
)
.await
}
/// Remove a stopped container without asking Podman to remove volumes.
///
/// This is deliberately separate from [`Self::remove_container`]: stopped
/// sandbox trust reconciliation replaces only the container and must never
/// remove its pre-existing named workspace volume.
pub async fn remove_container_preserving_volumes(
&self,
name: &str,
) -> Result<(), PodmanApiError> {
validate_name(name)?;
self.request_ok(
hyper::Method::DELETE,
&format!("/libpod/containers/{name}?force=true&volumes=false&timeout=0"),
None,
)
.await
}
/// Download a file from a container as a tar archive.
///
/// Calls `GET /libpod/containers/{name}/archive?path={path}` and returns
@@ -873,6 +913,30 @@ impl PodmanClient {
}
}
/// Check whether a named Podman secret exists without reading its value.
///
/// A 404 is an expected compatibility signal for legacy tokenless
/// sandboxes, so return it as `Ok(false)` without parsing or exposing the
/// response body. All other API failures remain errors.
pub async fn secret_exists(&self, name: &str) -> Result<bool, PodmanApiError> {
validate_name(name)?;
let (status, bytes) = self
.request(
hyper::Method::GET,
&format!("/libpod/secrets/{name}/json"),
None,
API_TIMEOUT,
)
.await?;
if status.is_success() {
Ok(true)
} else if status == hyper::StatusCode::NOT_FOUND {
Ok(false)
} else {
Err(error_from_response(status.as_u16(), &bytes))
}
}
/// Remove a Podman secret by name. Idempotent (not-found is ignored).
pub async fn remove_secret(&self, name: &str) -> Result<(), PodmanApiError> {
validate_name(name)?;
@@ -1341,6 +1405,58 @@ mod tests {
let _ = std::fs::remove_file(socket_path);
}
#[tokio::test]
async fn secret_exists_treats_not_found_as_absent_without_exposing_body() {
let (socket_path, request_log, handle) = spawn_podman_stub(
"secret-not-found",
vec![StubResponse::new(
StatusCode::NOT_FOUND,
"sensitive daemon diagnostic",
)],
);
let client = PodmanClient::new(socket_path.clone());
assert!(
!client
.secret_exists("openshell-token-sandbox-1")
.await
.expect("404 must be a normal absent-secret result")
);
handle.await.expect("stub should finish");
assert_eq!(
request_log
.lock()
.expect("request log lock should not be poisoned")
.as_slice(),
["GET /v5.0.0/libpod/secrets/openshell-token-sandbox-1/json"]
);
let _ = std::fs::remove_file(socket_path);
}
#[tokio::test]
async fn rename_container_uses_libpod_rename_endpoint() {
let (socket_path, request_log, handle) = spawn_podman_stub(
"rename-container",
vec![StubResponse::new(StatusCode::NO_CONTENT, "")],
);
let client = PodmanClient::new(socket_path.clone());
client
.rename_container("sandbox-123", "sandbox-123-rollback-1")
.await
.expect("container rename should succeed");
handle.await.expect("stub task should finish");
assert_eq!(
request_log
.lock()
.expect("request log lock should not be poisoned")
.as_slice(),
["POST /v5.0.0/libpod/containers/sandbox-123/rename?name=sandbox-123-rollback-1"]
);
let _ = std::fs::remove_file(socket_path);
}
#[tokio::test]
async fn remove_container_uses_single_timed_libpod_removal() {
let (socket_path, request_log, handle) = spawn_podman_stub(
crates/openshell-driver-podman/src/config.rs
+18
View File
@@ -203,6 +203,15 @@ pub struct PodmanComputeConfig {
/// system CA bundle. The bundle must exist and contain at least one
/// certificate; this is validated on the gateway host at sandbox-create
/// time, not deferred until the supervisor container starts.
///
/// Populated only from the `--sandbox-additional-ca-bundle` CLI argument
/// (or `OPENSHELL_SANDBOX_ADDITIONAL_CA_BUNDLE`) for standalone drivers.
/// Deliberately excluded from driver JSON config (`#[serde(skip)]` +
/// `deny_unknown_fields`): gateway-managed deployments must use the
/// global `[openshell.supervisor.network]` setting so every driver
/// receives the same staged bundle, and a per-driver path in operator
/// JSON is rejected instead of silently diverging from it.
#[serde(skip)]
pub additional_ca_bundle: Option<String>,
/// User namespace mode for sandbox containers (e.g. `auto`, `private`).
/// When unset, containers use the default user namespace.
@@ -649,6 +658,15 @@ mod tests {
assert!(error.to_string().contains("sandbox_ssh_socket_path"));
}
#[test]
fn config_rejects_driver_specific_destination_ca_setting() {
let error = serde_json::from_value::<PodmanComputeConfig>(serde_json::json!({
"additional_ca_bundle": ["/operator/ca.crt"]
}))
.expect_err("destination trust is global, not a Podman driver setting");
assert!(error.to_string().contains("additional_ca_bundle"));
}
#[test]
fn default_config_disables_health_checks() {
assert_eq!(
crates/openshell-driver-podman/src/container.rs
+218 -162
View File
@@ -10,9 +10,12 @@ use openshell_core::driver_mounts::SelinuxLabel;
use openshell_core::gpu::validate_cdi_device_names;
#[cfg(test)]
use openshell_core::gpu::{driver_gpu_requirements, validate_cdi_gpu_device_request};
use openshell_core::network_trust::{
NETWORK_SUPERVISOR_TRUST_GENERATION_KEY, NETWORK_SUPERVISOR_TRUST_GENERATION_NONE,
};
use openshell_core::proto::compute::v1::{DriverSandbox, DriverSandboxTemplate};
use openshell_core::proto_struct::deserialize_optional_non_empty_string_list;
use openshell_core::{driver_mounts, proto_struct};
use openshell_core::{NetworkSupervisorTrustBundle, driver_mounts, proto_struct};
use serde::Serialize;
use serde_json::Value;
use std::collections::{BTreeMap, HashSet};
@@ -613,17 +616,51 @@ fn upstream_proxy_cli_args(config: &PodmanComputeConfig) -> Vec<String> {
args
}
/// Build the additional-destination-CA command-line argument passed to the
/// supervisor. Independent of `upstream_proxy_cli_args`: this applies to
/// direct, inspected egress regardless of whether a corporate proxy is
/// configured.
fn additional_ca_cli_args(config: &PodmanComputeConfig) -> Vec<String> {
let mut args = Vec::new();
if config.additional_ca_bundle.is_some() {
args.push("--additional-ca-bundle".to_string());
args.push(ADDITIONAL_CA_MOUNT_PATH.to_string());
/// Build the destination-trust arguments for the trusted supervisor only.
///
/// The source path stays out of argv; the digest pairs the mounted artifact
/// with the immutable generation recorded on the workload label.
fn network_trust_cli_args(network_trust: Option<&NetworkSupervisorTrustBundle>) -> Vec<String> {
network_trust.map_or_else(Vec::new, |_bundle| {
vec![
"--additional-ca-bundle".to_string(),
ADDITIONAL_CA_MOUNT_PATH.to_string(),
]
})
}
fn network_trust_mount(bundle: &NetworkSupervisorTrustBundle) -> Result<Mount, ComputeDriverError> {
bundle.verify_artifact().map_err(|error| {
ComputeDriverError::Precondition(format!(
"verify gateway-owned network additional CA artifact: {error}"
))
})?;
let path = bundle.artifact_path();
let source = path.to_str().ok_or_else(|| {
ComputeDriverError::Precondition(format!(
"network additional CA artifact path is not valid UTF-8: {}",
path.display()
))
})?;
driver_mounts::validate_absolute_mount_source(source, "network additional CA artifact")
.map_err(ComputeDriverError::Precondition)?;
let mut options = vec!["ro".into(), "rbind".into()];
if is_selinux_enabled() {
options.push("z".into());
}
args
Ok(Mount {
kind: "bind".into(),
source: source.to_string(),
destination: ADDITIONAL_CA_MOUNT_PATH.into(),
options,
})
}
fn network_trust_generation(network_trust: Option<&NetworkSupervisorTrustBundle>) -> &str {
network_trust.map_or(NETWORK_SUPERVISOR_TRUST_GENERATION_NONE, |bundle| {
bundle.digest()
})
}
fn build_env(
@@ -1300,7 +1337,6 @@ fn build_base_spec(
image_volumes.extend(user_mounts.image_volumes);
let mut command = vec!["--workdir".to_string(), image.workspace_root.clone()];
command.extend(upstream_proxy_cli_args(config));
command.extend(additional_ca_cli_args(config));
let container_spec = ContainerSpec {
name,
@@ -1467,24 +1503,6 @@ fn build_base_spec(
options: ro,
});
}
// Bind-mount an operator-supplied additional destination CA
// bundle read-only when configured. Independent of the corporate
// proxy CA above: this applies to direct, inspected egress. The
// supervisor reads it via the --additional-ca-bundle argv path
// (see additional_ca_cli_args) and folds it into its upstream
// trust store alongside the system bundle.
if let Some(ca_bundle) = &config.additional_ca_bundle {
let mut ro = vec!["ro".into(), "rbind".into()];
if is_selinux_enabled() {
ro.push("z".into());
}
m.push(Mount {
kind: "bind".into(),
source: ca_bundle.clone(),
destination: ADDITIONAL_CA_MOUNT_PATH.into(),
options: ro,
});
}
if let Some(bin_path) = supervisor_bin_path {
let mut opts = vec!["ro".into(), "rbind".into()];
if is_selinux_enabled() {
@@ -1564,6 +1582,9 @@ pub struct IsolationSpecInput<'a> {
pub image: &'a ResolvedPodmanImage,
pub supervisor_bin: Option<&'a Path>,
pub tls_secrets: Option<&'a [String; 1]>,
/// Gateway-owned immutable destination-trust artifact. It is deliberately
/// mounted only into the trusted supervisor spec.
pub network_trust: Option<&'a NetworkSupervisorTrustBundle>,
pub identity: &'a openshell_isolation_interface::contract::ResolvedWorkloadIdentity,
}
@@ -1687,6 +1708,14 @@ pub fn build_isolation_specs(
supervisor
.labels
.insert(crate::isolation::LABEL_ROLE.into(), "supervisor".into());
// The workload is never replaced when the destination-trust bundle
// changes; only this supervisor is. Record the generation it was created
// with so a later start can detect a stale supervisor and recreate it
// without touching the workload or its data.
supervisor.labels.insert(
NETWORK_SUPERVISOR_TRUST_GENERATION_KEY.into(),
network_trust_generation(input.network_trust).into(),
);
supervisor.image.clone_from(&input.config.supervisor_image);
supervisor.entrypoint = vec!["/openshell-supervisor".into()];
supervisor.command.extend([
@@ -1697,6 +1726,12 @@ pub fn build_isolation_specs(
format!("--auth-bundle-file={}", crate::isolation::AUTH_BUNDLE_PATH),
"--health-socket-path=/run/openshell/supervisor-health.sock".into(),
]);
supervisor
.command
.extend(network_trust_cli_args(input.network_trust));
if let Some(network_trust) = input.network_trust {
supervisor.mounts.push(network_trust_mount(network_trust)?);
}
supervisor.env.insert(
openshell_core::sandbox_env::ADMITTED_ISOLATION_BACKEND.into(),
openshell_sandbox_backend::BACKEND_NAME.into(),
@@ -1919,6 +1954,122 @@ mod tests {
.expect("fixture image metadata should be valid")
}
#[test]
fn additional_destination_trust_is_delivered_only_to_supervisor_with_digest() {
let artifact = crate::test_utils::unique_socket_path("additional-ca").with_extension("crt");
let pem = b"normalized additional CA fixture";
std::fs::write(&artifact, pem).unwrap();
#[cfg(unix)]
std::fs::set_permissions(
&artifact,
std::os::unix::fs::PermissionsExt::from_mode(0o444),
)
.unwrap();
let bundle = NetworkSupervisorTrustBundle::new(
pem.to_vec(),
1,
"sha256:paired-digest",
artifact.clone(),
);
let sandbox = DriverSandbox {
id: "trust-pair".into(),
name: "trust-pair".into(),
..Default::default()
};
let identity = openshell_isolation_interface::contract::ResolvedWorkloadIdentity::new(
1000,
1001,
Vec::new(),
"image".into(),
"sha256:image".into(),
)
.unwrap();
let image = resolved_image("image", "sha256:image", "1000:1001", "/workspace/project");
let specs = build_isolation_specs(IsolationSpecInput {
sandbox: &sandbox,
config: &PodmanComputeConfig::default(),
token_secret: None,
resolver_secret: "resolver",
gpu_devices: None,
image: &image,
supervisor_bin: None,
tls_secrets: None,
network_trust: Some(&bundle),
identity: &identity,
})
.unwrap();
assert_eq!(
specs
.supervisor
.labels
.get(NETWORK_SUPERVISOR_TRUST_GENERATION_KEY)
.map(String::as_str),
Some("sha256:paired-digest")
);
assert!(
specs
.workload
.command
.iter()
.all(|argument| argument != "--additional-ca-bundle")
);
assert!(
specs
.workload
.mounts
.iter()
.all(|mount| { mount.destination != ADDITIONAL_CA_MOUNT_PATH })
);
assert!(specs.supervisor.command.windows(2).any(|arguments| {
arguments == ["--additional-ca-bundle", ADDITIONAL_CA_MOUNT_PATH]
}));
assert!(specs.supervisor.mounts.iter().any(|mount| {
mount.destination == ADDITIONAL_CA_MOUNT_PATH
&& mount.source == artifact.to_string_lossy()
&& mount.options.iter().any(|option| option == "ro")
}));
let _ = std::fs::remove_file(artifact);
}
#[test]
fn tampered_destination_trust_artifact_fails_closed_without_pem_contents() {
let artifact =
crate::test_utils::unique_socket_path("tampered-additional-ca").with_extension("crt");
std::fs::write(&artifact, b"expected").unwrap();
#[cfg(unix)]
std::fs::set_permissions(
&artifact,
std::os::unix::fs::PermissionsExt::from_mode(0o444),
)
.unwrap();
let bundle = NetworkSupervisorTrustBundle::new(
b"expected".to_vec(),
1,
"sha256:expected",
artifact.clone(),
);
std::fs::remove_file(&artifact).unwrap();
std::fs::write(&artifact, b"BEGIN CERTIFICATE tampered contents").unwrap();
#[cfg(unix)]
std::fs::set_permissions(
&artifact,
std::os::unix::fs::PermissionsExt::from_mode(0o444),
)
.unwrap();
let Err(error) = network_trust_mount(&bundle) else {
panic!("tampering must fail closed");
};
assert!(
error
.to_string()
.contains("gateway-owned network additional CA artifact")
);
assert!(!error.to_string().contains("tampered contents"));
let _ = std::fs::remove_file(artifact);
}
#[test]
fn isolated_pair_keeps_privileges_network_and_secrets_out_of_workload() {
let sandbox = DriverSandbox {
@@ -1953,6 +2104,7 @@ mod tests {
image: &image,
supervisor_bin: None,
tls_secrets: None,
network_trust: None,
identity: &identity,
})
.unwrap();
@@ -1974,6 +2126,14 @@ mod tests {
assert_eq!(specs.supervisor.groups, vec!["2000"]);
assert!(specs.supervisor.cap_add.is_empty());
assert_eq!(specs.workload.netns.nsmode, "none");
assert_eq!(
specs
.supervisor
.labels
.get(NETWORK_SUPERVISOR_TRUST_GENERATION_KEY)
.map(String::as_str),
Some(NETWORK_SUPERVISOR_TRUST_GENERATION_NONE)
);
assert_eq!(
specs
.workload
@@ -2006,6 +2166,7 @@ mod tests {
image: &ResolvedPodmanImage::new("sha256:image"),
supervisor_bin: None,
tls_secrets: None,
network_trust: None,
identity: &default_identity,
})
.unwrap();
@@ -2961,127 +3122,17 @@ mod tests {
),
"no CA bundle mount without operator config"
);
}
#[test]
fn container_spec_binds_additional_ca_bundle_and_passes_argv_without_proxy() {
// #3781: the additional CA must work for direct, inspected egress
// with no corporate proxy configured at all — unlike proxy_ca_bundle,
// which requires https_proxy.
let sandbox = test_sandbox("additional-ca-id", "additional-ca-name");
let mut config = test_config();
assert!(
config.https_proxy.is_none(),
"test must exercise the no-proxy case"
);
config.additional_ca_bundle = Some("/host/additional-ca.pem".to_string());
let spec = build_container_spec(&sandbox, &config);
let command = spec_command(&spec);
let idx = command
.iter()
.position(|a| a == "--additional-ca-bundle")
.expect("additional CA bundle flag present");
assert_eq!(
command.get(idx + 1).map(String::as_str),
Some("/etc/openshell/tls/additional/ca-bundle.pem")
);
let mounts = spec["mounts"]
.as_array()
.expect("mounts should be an array");
let ca_mount = mounts
.iter()
.find(|m| {
m["type"].as_str() == Some("bind")
&& m["destination"].as_str()
== Some("/etc/openshell/tls/additional/ca-bundle.pem")
})
.expect("additional CA bundle bind mount present");
assert_eq!(ca_mount["source"].as_str(), Some("/host/additional-ca.pem"));
assert!(
ca_mount["options"]
.as_array()
.expect("options array")
!mounts
.iter()
.any(|o| o.as_str() == Some("ro")),
"additional CA bundle mount must be read-only"
);
}
#[test]
fn isolated_pair_mounts_additional_ca_bundle_only_on_supervisor() {
// build_isolation_specs() filters the base spec's mounts per role, so
// the base-spec test above does not prove the final supervisor keeps
// the CA the --additional-ca-bundle argv points at.
let sandbox = test_sandbox("additional-ca-pair", "additional-ca-pair");
let mut config = test_config();
config.additional_ca_bundle = Some("/host/additional-ca.pem".to_string());
let identity = openshell_isolation_interface::contract::ResolvedWorkloadIdentity::new(
1000,
1000,
Vec::new(),
"default".into(),
"sha256:image".into(),
)
.unwrap();
let specs = build_isolation_specs(IsolationSpecInput {
sandbox: &sandbox,
config: &config,
token_secret: Some("jwt"),
resolver_secret: "resolver",
gpu_devices: None,
image: &ResolvedPodmanImage::new("sha256:image"),
supervisor_bin: None,
tls_secrets: None,
identity: &identity,
})
.unwrap();
let ca_mount = specs
.supervisor
.mounts
.iter()
.find(|mount| mount.destination == ADDITIONAL_CA_MOUNT_PATH)
.expect("supervisor keeps the additional CA bind mount");
assert_eq!(ca_mount.kind, "bind");
assert_eq!(ca_mount.source, "/host/additional-ca.pem");
assert!(ca_mount.options.contains(&"ro".to_string()));
let idx = specs
.supervisor
.command
.iter()
.position(|arg| arg == "--additional-ca-bundle")
.expect("supervisor argv carries the additional CA flag");
assert_eq!(
specs.supervisor.command.get(idx + 1).map(String::as_str),
Some(ADDITIONAL_CA_MOUNT_PATH)
.any(|m| { m["destination"].as_str() == Some(ADDITIONAL_CA_MOUNT_PATH) }),
"no destination CA mount without global config"
);
assert!(
specs
.workload
.mounts
!spec_command(&spec)
.iter()
.all(|mount| mount.destination != ADDITIONAL_CA_MOUNT_PATH),
"workload must not receive the additional CA mount"
);
}
#[test]
fn container_spec_omits_additional_ca_bundle_when_unconfigured() {
let sandbox = test_sandbox("test-id", "test-name");
let spec = build_container_spec(&sandbox, &test_config());
let command = spec_command(&spec);
assert!(
!command.iter().any(|a| a == "--additional-ca-bundle"),
"no additional-ca-bundle flag without operator config"
);
let mounts = spec["mounts"].as_array().expect("mounts array");
assert!(
!mounts.iter().any(|m| m["destination"].as_str()
== Some("/etc/openshell/tls/additional/ca-bundle.pem")),
"no additional CA bundle mount without operator config"
.any(|arg| arg == "--additional-ca-bundle"),
"no destination CA argument without global config"
);
}
@@ -3766,25 +3817,30 @@ mod tests {
fn driver_config_rejects_reserved_mount_targets() {
use openshell_core::proto::compute::v1::{DriverSandboxSpec, DriverSandboxTemplate};
let mut sandbox = test_sandbox("test-id", "test-name");
sandbox.spec = Some(DriverSandboxSpec {
template: Some(DriverSandboxTemplate {
driver_config: Some(json_struct(serde_json::json!({
"mounts": [{
"type": "volume",
"source": "work-nfs",
"target": "/etc/openshell/tls/client"
}]
}))),
for target in ["/etc/openshell/tls/client", ADDITIONAL_CA_MOUNT_PATH] {
let mut sandbox = test_sandbox("test-id", "test-name");
sandbox.spec = Some(DriverSandboxSpec {
template: Some(DriverSandboxTemplate {
driver_config: Some(json_struct(serde_json::json!({
"mounts": [{
"type": "volume",
"source": "work-nfs",
"target": target
}]
}))),
..Default::default()
}),
..Default::default()
}),
..Default::default()
});
let config = test_config();
});
let config = test_config();
let err = try_build_container_spec_with_token(&sandbox, &config, None).unwrap_err();
let err = try_build_container_spec_with_token(&sandbox, &config, None).unwrap_err();
assert!(err.to_string().contains("reserved OpenShell path"));
assert!(
err.to_string().contains("reserved OpenShell path"),
"target {target}: {err}"
);
}
}
#[test]
crates/openshell-driver-podman/src/driver.rs
+829 -69
View File
File diff suppressed because it is too large Load Diff
crates/openshell-driver-podman/src/grpc.rs
+20 -17
View File
@@ -187,23 +187,26 @@ impl ComputeDriver for ComputeDriverService {
&self,
request: Request<StartSandboxRequest>,
) -> Result<Response<StartSandboxResponse>, Status> {
self.rpc_tracer
.trace(openshell_otel::rpc::START_SANDBOX, async {
let request = request.into_inner();
if request.sandbox_id.is_empty() {
return Err(Status::invalid_argument("sandbox_id is required"));
}
self.driver
.start_sandbox(
&request.sandbox_id,
&request.generation_id,
&request.launch_authentication,
)
.await
.map_err(Status::from)?;
Ok(Response::new(StartSandboxResponse::default()))
})
.await
Box::pin(
self.rpc_tracer
.trace(openshell_otel::rpc::START_SANDBOX, async {
let request = request.into_inner();
if request.sandbox_id.is_empty() {
return Err(Status::invalid_argument("sandbox_id is required"));
}
self.driver
.start_sandbox(
&request.sandbox_id,
&request.generation_id,
&request.launch_authentication,
request.sandbox.as_ref(),
)
.await
.map_err(Status::from)?;
Ok(Response::new(StartSandboxResponse::default()))
}),
)
.await
}
async fn delete_sandbox(
crates/openshell-driver-podman/src/main.rs
+52 -47
View File
@@ -227,53 +227,58 @@ async fn main() -> Result<()> {
},
);
let driver = PodmanComputeDriver::new(PodmanComputeConfig {
allow_driver_config: args.admission_config_json.allow_driver_config,
resource_admission: args.admission_config_json.resource_admission.clone(),
socket_path: args.podman_socket,
default_image: args.sandbox_image.unwrap_or_default(),
image_pull_policy: args.sandbox_image_pull_policy,
grpc_endpoint: args.grpc_endpoint.unwrap_or_default(),
supervisor_otlp_endpoint: args.otlp_endpoint.clone(),
supervisor_gateway_name: args.gateway_name.clone(),
supervisor_ocsf_export: args.otlp_ocsf_endpoint.clone().map(|endpoint| {
openshell_core::sandbox_env::OcsfOtlpExport {
endpoint,
schema_version: args.otlp_ocsf_schema_version.clone(),
}
}),
gateway_port: args.gateway_port,
host_gateway_ip: args
.host_gateway_ip
.unwrap_or_else(PodmanComputeConfig::default_host_gateway_ip),
ssh_socket_path: args.sandbox_ssh_socket_path,
network_name: args.network_name,
stop_timeout_secs: args.stop_timeout,
sandbox_runtime_image: args
.sandbox_runtime_image
.unwrap_or_else(openshell_core::config::default_sandbox_runtime_image),
supervisor_image: args
.supervisor_image
.unwrap_or_else(openshell_core::config::default_supervisor_image),
guest_tls_ca: args.podman_tls_ca,
guest_tls_cert: args.podman_tls_cert,
guest_tls_key: args.podman_tls_key,
provider_spiffe_workload_api_socket: args.provider_spiffe_workload_api_socket,
app_armor_profile: args.app_armor_profile,
sandbox_pids_limit: args.sandbox_pids_limit,
health_check_interval_secs: args.health_check_interval_secs,
https_proxy: args.sandbox_https_proxy,
no_proxy: args.sandbox_no_proxy,
proxy_auth_file: args.sandbox_proxy_auth_file,
proxy_auth_allow_insecure: args.sandbox_proxy_auth_allow_insecure,
proxy_connect_by_hostname: args.sandbox_proxy_connect_by_hostname,
proxy_ca_bundle: args.sandbox_proxy_ca_bundle,
additional_ca_bundle: args.sandbox_additional_ca_bundle,
userns: args.userns,
uidmap: args.uidmap,
gidmap: args.gidmap,
enable_bind_mounts: args.enable_bind_mounts,
})
// External drivers have no gateway-local staged artifact. Do not accept
// a path or silently emulate destination-CA delivery outside the gateway.
let driver = PodmanComputeDriver::new(
PodmanComputeConfig {
allow_driver_config: args.admission_config_json.allow_driver_config,
resource_admission: args.admission_config_json.resource_admission.clone(),
socket_path: args.podman_socket,
default_image: args.sandbox_image.unwrap_or_default(),
image_pull_policy: args.sandbox_image_pull_policy,
grpc_endpoint: args.grpc_endpoint.unwrap_or_default(),
supervisor_otlp_endpoint: args.otlp_endpoint.clone(),
supervisor_gateway_name: args.gateway_name.clone(),
supervisor_ocsf_export: args.otlp_ocsf_endpoint.clone().map(|endpoint| {
openshell_core::sandbox_env::OcsfOtlpExport {
endpoint,
schema_version: args.otlp_ocsf_schema_version.clone(),
}
}),
gateway_port: args.gateway_port,
host_gateway_ip: args
.host_gateway_ip
.unwrap_or_else(PodmanComputeConfig::default_host_gateway_ip),
ssh_socket_path: args.sandbox_ssh_socket_path,
network_name: args.network_name,
stop_timeout_secs: args.stop_timeout,
sandbox_runtime_image: args
.sandbox_runtime_image
.unwrap_or_else(openshell_core::config::default_sandbox_runtime_image),
supervisor_image: args
.supervisor_image
.unwrap_or_else(openshell_core::config::default_supervisor_image),
guest_tls_ca: args.podman_tls_ca,
guest_tls_cert: args.podman_tls_cert,
guest_tls_key: args.podman_tls_key,
provider_spiffe_workload_api_socket: args.provider_spiffe_workload_api_socket,
app_armor_profile: args.app_armor_profile,
sandbox_pids_limit: args.sandbox_pids_limit,
health_check_interval_secs: args.health_check_interval_secs,
https_proxy: args.sandbox_https_proxy,
no_proxy: args.sandbox_no_proxy,
proxy_auth_file: args.sandbox_proxy_auth_file,
proxy_auth_allow_insecure: args.sandbox_proxy_auth_allow_insecure,
proxy_connect_by_hostname: args.sandbox_proxy_connect_by_hostname,
proxy_ca_bundle: args.sandbox_proxy_ca_bundle,
additional_ca_bundle: args.sandbox_additional_ca_bundle,
userns: args.userns,
uidmap: args.uidmap,
gidmap: args.gidmap,
enable_bind_mounts: args.enable_bind_mounts,
},
None,
)
.await
.into_diagnostic()?;
crates/openshell-driver-podman/src/watcher.rs
+28 -1
View File
@@ -681,17 +681,44 @@ mod tests {
}
fn podman_event(action: &str, sandbox_id: &str, time_nano: i64) -> PodmanEvent {
podman_event_for_container(action, sandbox_id, "container-1", time_nano)
}
fn podman_event_for_container(
action: &str,
sandbox_id: &str,
container_id: &str,
time_nano: i64,
) -> PodmanEvent {
PodmanEvent {
event_type: "container".to_string(),
action: action.to_string(),
actor: crate::client::EventActor {
id: "container-1".to_string(),
id: container_id.to_string(),
attributes: HashMap::from([(LABEL_SANDBOX_ID.to_string(), sandbox_id.to_string())]),
},
time_nano,
}
}
#[tokio::test]
async fn unrelated_remove_deletes_the_sandbox() {
let fences = LifecycleEventFences::default();
let client = PodmanClient::new(crate::test_utils::unique_socket_path("watch-removals"));
let event = map_podman_event(
&podman_event_for_container("remove", "sandbox-1", "external-container", 2),
&client,
&fences,
)
.await
.expect("an unrelated remove must still delete the sandbox");
assert!(matches!(
event.payload,
Some(watch_sandboxes_event::Payload::Deleted(_))
));
}
#[test]
fn lifecycle_fence_rejects_delayed_stop_events_from_before_restart() {
let fences = LifecycleEventFences::default();
crates/openshell-driver-vm/Cargo.toml
+1
View File
@@ -118,6 +118,7 @@ defaults-without-telemetry = ["compute-driver"]
openshell-otel-test-support = { path = "../openshell-otel-test-support" }
temp-env = "0.3"
tempfile = "3"
rcgen = { workspace = true }
opentelemetry_sdk = { workspace = true, features = ["testing"] }
# smol-rs/polling drives the BSD/macOS parent-death detection in
crates/openshell-driver-vm/src/driver.rs
+730 -6
View File
@@ -27,6 +27,7 @@ use crate::rootfs::{
validate_host_supervisor, write_rootfs_image_file,
};
use crate::runtime::VmBackend;
use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD};
use bollard::Docker;
use bollard::errors::Error as BollardError;
use bollard::models::ContainerCreateBody;
@@ -71,6 +72,9 @@ use openshell_core::proto::compute::v1::{
use openshell_core::proto_struct::{
deserialize_optional_non_empty_string_list, struct_to_json_value,
};
use openshell_core::{
NetworkSupervisorTrustBundle, network_trust::MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES,
};
use openshell_sandbox_backend::boundary_protocol::{
BoundaryConfig, BoundaryListener, GatewayVerificationKey, SandboxRuntimeDescriptor,
SandboxTlsClientConfig, SandboxTlsMaterial, SandboxTlsServerConfig, SandboxTransport,
@@ -86,11 +90,9 @@ use std::fs;
use std::future::Future;
use std::io::{BufRead, BufReader, BufWriter, Read, Seek, SeekFrom, Write};
#[cfg(unix)]
use std::os::fd::AsRawFd as _;
use std::os::fd::{AsRawFd as _, OwnedFd};
#[cfg(unix)]
use std::os::fd::OwnedFd;
#[cfg(unix)]
use std::os::unix::fs::PermissionsExt;
use std::os::unix::fs::{OpenOptionsExt as _, PermissionsExt};
#[cfg(unix)]
use std::os::unix::process::ExitStatusExt as _;
use std::path::{Component, Path, PathBuf};
@@ -213,6 +215,15 @@ const GUEST_BOUNDARY_CONFIG_ENV: &str = "OPENSHELL_VM_SANDBOX_BOOTSTRAP";
const HOST_AUTH_BUNDLE_FILE: &str = "supervisor-auth.json";
const HOST_RUNTIME_DESCRIPTOR_FILE: &str = "runtime-descriptor.json";
const HOST_BOUNDARY_GENERATION_FILE: &str = "boundary-generation";
/// Per-sandbox host-only copy of the gateway-normalized destination trust.
///
/// This file is deliberately outside the VM disks. The host supervisor is the
/// only process which receives its path; the guest runs `openshell-sandbox`
/// and must never receive this material through its overlay, rootfs, argv, or
/// environment.
const HOST_ADDITIONAL_CA_FILE: &str = "additional-ca.crt";
const HOST_ADDITIONAL_CA_TEMP_PREFIX: &str = ".additional-ca.";
const HOST_ADDITIONAL_CA_TEMP_SUFFIX: &str = ".tmp";
/// The backend this driver admits. VM-specific placement remains inside the
/// opaque runtime descriptor.
const DRIVER_ADMITTED_BACKEND: &str = openshell_sandbox_backend::BACKEND_NAME;
@@ -257,6 +268,7 @@ const IMAGE_PREP_CONSOLE_LOG: &str = "image-prep-console.log";
const PREPARED_IMAGE_ROOTFS_DIR: &str = "/image-rootfs";
static IMAGE_CACHE_BUILD_COUNTER: AtomicU64 = AtomicU64::new(0);
static OWNER_STATE_WRITE_COUNTER: AtomicU64 = AtomicU64::new(0);
static NETWORK_TRUST_WRITE_COUNTER: AtomicU64 = AtomicU64::new(0);
#[derive(Debug, Clone)]
struct VmDriverTlsPaths {
@@ -290,6 +302,41 @@ enum GuestImagePayloadSource {
LocalDocker { rootfs_archive: PathBuf },
}
/// Driver-owned immutable copy of gateway-normalized destination trust.
///
/// This is public only because [`VmDriverConfig`] is public. Its contents are
/// intentionally not constructible outside this module, and
/// [`VmDriver::new`] rejects a value supplied by a caller. The VM driver alone
/// creates it after reading and verifying the gateway artifact once.
#[derive(Clone)]
pub struct VmNetworkSupervisorTrustSnapshot {
bundle: NetworkSupervisorTrustBundle,
}
impl VmNetworkSupervisorTrustSnapshot {
fn normalized_pem(&self) -> &[u8] {
self.bundle.normalized_pem()
}
fn digest(&self) -> &str {
self.bundle.digest()
}
fn certificate_count(&self) -> usize {
self.bundle.certificate_count()
}
}
/// Redact PEM bytes and their original gateway artifact path from diagnostics.
impl std::fmt::Debug for VmNetworkSupervisorTrustSnapshot {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("VmNetworkSupervisorTrustSnapshot")
.field("certificate_count", &self.certificate_count())
.field("digest", &self.digest())
.finish_non_exhaustive()
}
}
#[derive(Clone, serde::Serialize, serde::Deserialize)]
#[serde(deny_unknown_fields)]
#[allow(clippy::struct_excessive_bools)]
@@ -323,7 +370,22 @@ pub struct VmDriverConfig {
pub guest_tls_ca: Option<PathBuf>,
pub guest_tls_cert: Option<PathBuf>,
pub guest_tls_key: Option<PathBuf>,
/// Corporate forward proxy settings delivered to the guest init script.
/// Gateway-owned, normalized destination-trust artifact path received on
/// the private gateway-to-driver argv. It is consumed exactly once during
/// driver initialization and then cleared.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub additional_ca_bundle: Option<PathBuf>,
/// SHA-256 generation expected for `additional_ca_bundle`.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub additional_ca_digest: Option<String>,
/// Number of canonical certificate blocks expected in the artifact.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub additional_ca_certificate_count: Option<usize>,
/// Internal immutable trust state. Callers must leave this unset; it is
/// deliberately excluded from all serialized configuration input/output.
#[serde(skip)]
pub additional_ca_snapshot: Option<VmNetworkSupervisorTrustSnapshot>,
/// Corporate forward proxy settings delivered to the host supervisor.
#[serde(flatten)]
pub upstream_proxy: UpstreamProxyConfig,
/// Gateway-host PEM CA bundle staged into the guest overlay for the
@@ -382,6 +444,10 @@ impl std::fmt::Debug for VmDriverConfig {
.field("guest_tls_ca", &self.guest_tls_ca)
.field("guest_tls_cert", &self.guest_tls_cert)
.field("guest_tls_key", &self.guest_tls_key)
.field(
"additional_ca_configured",
&(self.additional_ca_bundle.is_some() || self.additional_ca_snapshot.is_some()),
)
.field("gpu_enabled", &self.gpu_enabled)
.field("gpu_mem_mib", &self.gpu_mem_mib)
.field("gpu_vcpus", &self.gpu_vcpus)
@@ -452,6 +518,10 @@ impl Default for VmDriverConfig {
guest_tls_ca: None,
guest_tls_cert: None,
guest_tls_key: None,
additional_ca_bundle: None,
additional_ca_digest: None,
additional_ca_certificate_count: None,
additional_ca_snapshot: None,
upstream_proxy: UpstreamProxyConfig::default(),
proxy_ca_bundle: None,
provider_spiffe_workload_api_tcp_endpoint: None,
@@ -598,6 +668,353 @@ impl VmDriverConfig {
}
}
/// Consume the gateway's private launch metadata and retain only a
/// driver-owned immutable trust snapshot.
///
/// The standalone VM driver cannot receive the bundle bytes over the gateway
/// RPC, so the in-process gateway passes a path, digest, and count on its
/// authenticated child-process argv. Treat those three values as one atomic
/// contract: accepting a partial set would make it possible to read a mutable
/// path without an authenticated generation or to silently lose a trust
/// anchor. The source is read only here; launches use the resulting bytes and
/// never re-open the gateway artifact.
fn initialize_additional_ca_snapshot(config: &mut VmDriverConfig) -> Result<(), String> {
if config.additional_ca_snapshot.is_some() {
return Err(
"additional CA snapshot is driver-owned and must not be supplied by callers"
.to_string(),
);
}
let snapshot = match (
config.additional_ca_bundle.as_ref(),
config.additional_ca_digest.as_deref(),
config.additional_ca_certificate_count,
) {
(None, None, None) => None,
(Some(path), Some(expected_digest), Some(expected_certificate_count)) => {
if path.as_os_str().is_empty() {
return Err("additional CA artifact path must not be empty".to_string());
}
if expected_certificate_count == 0 {
return Err("additional CA certificate count must be greater than zero".to_string());
}
let expected_digest = normalized_additional_ca_digest(expected_digest)?;
let pem = read_gateway_additional_ca_artifact(path)?;
// Deterministic destination-trust generation label, not an
// authenticity proof: the equality check below only detects
// staged-artifact drift; trust comes from the gateway's
// normalization and TLS verification. Classified as a non-security
// identity hash in openshell-crypto/README.md.
let actual_digest = format!("sha256:{:x}", Sha256::digest(&pem));
if actual_digest != expected_digest {
return Err(format!(
"additional CA artifact '{}' does not match expected digest {}",
path.display(),
expected_digest
));
}
let actual_certificate_count = normalized_additional_ca_certificate_count(&pem)
.map_err(|error| {
format!(
"additional CA artifact '{}' is not a normalized certificate bundle: {error}",
path.display()
)
})?;
if actual_certificate_count != expected_certificate_count {
return Err(format!(
"additional CA artifact '{}' has certificate count {}, expected {}",
path.display(),
actual_certificate_count,
expected_certificate_count
));
}
Some(VmNetworkSupervisorTrustSnapshot {
// The bundle's artifact path records provenance only. Never
// call `verify_artifact` on this retained value: all later
// launches must use these captured immutable bytes rather
// than rereading the mutable gateway-host source path.
bundle: NetworkSupervisorTrustBundle::new(
pem,
actual_certificate_count,
expected_digest,
path.clone(),
),
})
}
_ => {
return Err(
"additional CA artifact path, digest, and certificate count must be supplied together"
.to_string(),
);
}
};
// Do not retain the mutable gateway artifact path or metadata after the
// authenticated, bounded startup read. In particular this makes a later
// sandbox launch incapable of accidentally rereading it.
config.additional_ca_bundle = None;
config.additional_ca_digest = None;
config.additional_ca_certificate_count = None;
config.additional_ca_snapshot = snapshot;
Ok(())
}
/// Require the generation form emitted by gateway normalization.
fn normalized_additional_ca_digest(digest: &str) -> Result<String, String> {
let Some(hex) = digest.strip_prefix("sha256:") else {
return Err("additional CA digest must use the sha256: prefix".to_string());
};
if hex.len() != 64 || !hex.bytes().all(|byte| byte.is_ascii_hexdigit()) {
return Err("additional CA digest must be a SHA-256 digest".to_string());
}
if hex.bytes().any(|byte| byte.is_ascii_uppercase()) {
return Err("additional CA digest must use normalized lowercase hexadecimal".to_string());
}
Ok(format!("sha256:{hex}"))
}
/// Read a normalized gateway artifact once without following symlinks.
fn read_gateway_additional_ca_artifact(path: &Path) -> Result<Vec<u8>, String> {
let metadata = fs::symlink_metadata(path).map_err(|error| {
format!(
"inspect additional CA artifact '{}': {error}",
path.display()
)
})?;
if !metadata.file_type().is_file() {
return Err(format!(
"additional CA artifact '{}' is not a regular file",
path.display()
));
}
validate_gateway_additional_ca_artifact_metadata(path, &metadata)?;
let mut options = fs::OpenOptions::new();
options.read(true);
#[cfg(unix)]
{
// The pre-open metadata check gives portable diagnostics, while these
// flags ensure an attacker cannot replace it with a symlink, FIFO, or
// blocking special file before the actual one-time read.
options.custom_flags(libc::O_CLOEXEC | libc::O_NOFOLLOW | libc::O_NONBLOCK);
}
let mut file = options
.open(path)
.map_err(|error| format!("open additional CA artifact '{}': {error}", path.display()))?;
let opened_metadata = file.metadata().map_err(|error| {
format!(
"inspect additional CA artifact '{}': {error}",
path.display()
)
})?;
if !opened_metadata.file_type().is_file() {
return Err(format!(
"additional CA artifact '{}' is not a regular file",
path.display()
));
}
validate_gateway_additional_ca_artifact_metadata(path, &opened_metadata)?;
let capacity = usize::try_from(opened_metadata.len()).map_err(|_| {
format!(
"additional CA artifact '{}' exceeds the {}-byte limit",
path.display(),
MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES
)
})?;
let mut pem = Vec::with_capacity(capacity);
Read::by_ref(&mut file)
.take((MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES + 1) as u64)
.read_to_end(&mut pem)
.map_err(|error| format!("read additional CA artifact '{}': {error}", path.display()))?;
if pem.len() > MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES {
return Err(format!(
"additional CA artifact '{}' exceeds the {}-byte limit",
path.display(),
MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES
));
}
Ok(pem)
}
fn validate_gateway_additional_ca_artifact_metadata(
path: &Path,
metadata: &fs::Metadata,
) -> Result<(), String> {
if metadata.len() > MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES as u64 {
return Err(format!(
"additional CA artifact '{}' exceeds the {}-byte limit",
path.display(),
MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES
));
}
#[cfg(unix)]
if metadata.permissions().mode() & 0o7777 != 0o444 {
return Err(format!(
"additional CA artifact '{}' does not have the required read-only permissions",
path.display()
));
}
Ok(())
}
/// Verify that the bytes are exactly the canonical PEM representation emitted
/// by the gateway's destination-trust normalization boundary, and return its
/// number of certificate blocks.
fn normalized_additional_ca_certificate_count(pem: &[u8]) -> Result<usize, &'static str> {
let text = std::str::from_utf8(pem).map_err(|_| "bundle is not UTF-8 PEM text")?;
let mut remainder = text;
let mut normalized = Vec::with_capacity(pem.len());
let mut certificate_count = 0_usize;
while !remainder.is_empty() {
let Some(after_begin) = remainder.strip_prefix("-----BEGIN CERTIFICATE-----\n") else {
return Err("bundle contains material outside canonical certificate blocks");
};
remainder = after_begin;
let mut encoded = String::new();
loop {
let Some(newline) = remainder.find('\n') else {
return Err("certificate block is missing its terminating newline");
};
let line = &remainder[..newline];
remainder = &remainder[newline + 1..];
if line == "-----END CERTIFICATE-----" {
break;
}
if line.is_empty() || line.len() > 64 {
return Err("certificate block has a non-canonical base64 line");
}
encoded.push_str(line);
}
let decoded = BASE64_STANDARD
.decode(encoded.as_bytes())
.map_err(|_| "certificate block contains invalid base64")?;
if decoded.is_empty() {
return Err("certificate block is empty");
}
normalized.extend_from_slice(b"-----BEGIN CERTIFICATE-----\n");
let canonical = BASE64_STANDARD.encode(decoded);
for line in canonical.as_bytes().chunks(64) {
normalized.extend_from_slice(line);
normalized.push(b'\n');
}
normalized.extend_from_slice(b"-----END CERTIFICATE-----\n");
certificate_count += 1;
}
if certificate_count == 0 {
return Err("bundle contains no certificate blocks");
}
if normalized != pem {
return Err("bundle is not canonical normalized PEM");
}
Ok(certificate_count)
}
/// Write or remove the host-only per-sandbox trust artifact.
///
/// The snapshot has already authenticated the bytes against the gateway's
/// generation. Do not replace this with a source-path copy: re-opening that
/// path here would let a changed gateway artifact alter later sandbox
/// launches. The temporary file is created with restrictive permissions and
/// atomically published into the sandbox's private state directory.
fn stage_additional_ca_snapshot(
state_dir: &Path,
snapshot: Option<&VmNetworkSupervisorTrustSnapshot>,
) -> Result<Option<PathBuf>, String> {
let destination = state_dir.join(HOST_ADDITIONAL_CA_FILE);
let Some(snapshot) = snapshot else {
remove_additional_ca_material(state_dir, &destination)?;
return Ok(None);
};
let temporary = state_dir.join(format!(
"{HOST_ADDITIONAL_CA_TEMP_PREFIX}{}-{}{}",
std::process::id(),
NETWORK_TRUST_WRITE_COUNTER.fetch_add(1, Ordering::Relaxed),
HOST_ADDITIONAL_CA_TEMP_SUFFIX,
));
let result: Result<(), String> = (|| {
let mut options = fs::OpenOptions::new();
options.write(true).create_new(true);
#[cfg(unix)]
options.mode(0o600);
let mut file = options
.open(&temporary)
.map_err(|error| format!("create host additional CA temporary file: {error}"))?;
file.write_all(snapshot.normalized_pem())
.map_err(|error| format!("write host additional CA temporary file: {error}"))?;
file.sync_all()
.map_err(|error| format!("sync host additional CA temporary file: {error}"))?;
#[cfg(unix)]
fs::set_permissions(&temporary, fs::Permissions::from_mode(0o600))
.map_err(|error| format!("restrict host additional CA temporary file: {error}"))?;
fs::rename(&temporary, &destination)
.map_err(|error| format!("publish host additional CA file: {error}"))?;
fs::File::open(state_dir)
.and_then(|directory| directory.sync_all())
.map_err(|error| format!("sync host additional CA directory: {error}"))?;
Ok(())
})();
if result.is_err() {
let _ = fs::remove_file(&temporary);
}
result?;
Ok(Some(destination))
}
/// Remove both the published host artifact and interrupted-write remnants.
fn remove_additional_ca_material(state_dir: &Path, destination: &Path) -> Result<(), String> {
remove_additional_ca_file(destination)?;
let entries = fs::read_dir(state_dir)
.map_err(|error| format!("read sandbox state while clearing additional CA: {error}"))?;
for entry in entries {
let entry = entry.map_err(|error| {
format!("read sandbox state entry while clearing additional CA: {error}")
})?;
let name = entry.file_name();
let name = name.to_string_lossy();
if name.starts_with(HOST_ADDITIONAL_CA_TEMP_PREFIX)
&& name.ends_with(HOST_ADDITIONAL_CA_TEMP_SUFFIX)
{
remove_additional_ca_file(&entry.path())?;
}
}
Ok(())
}
fn remove_additional_ca_file(path: &Path) -> Result<(), String> {
match fs::remove_file(path) {
Ok(()) => Ok(()),
Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(()),
Err(error) => Err(format!(
"remove host additional CA file '{}': {error}",
path.display()
)),
}
}
/// Render the host-supervisor-only arguments for a staged trust snapshot.
///
/// PEM bytes never appear in argv. The supervisor receives only the
/// restrictive host file path and gateway-authenticated generation digest.
fn additional_ca_supervisor_args(
snapshot: Option<&VmNetworkSupervisorTrustSnapshot>,
staged_path: Option<&Path>,
) -> Result<Vec<String>, String> {
match (snapshot, staged_path) {
(None, None) => Ok(Vec::new()),
(Some(_snapshot), Some(path)) => Ok(vec![
"--additional-ca-bundle".to_string(),
path.display().to_string(),
]),
(Some(_), None) => Err("additional CA snapshot was not staged".to_string()),
(None, Some(_)) => Err("staged additional CA exists without a snapshot".to_string()),
}
}
fn validate_openshell_endpoint(endpoint: &str) -> Result<(), String> {
let url = Url::parse(endpoint)
.map_err(|err| format!("invalid openshell endpoint '{endpoint}': {err}"))?;
@@ -766,6 +1183,7 @@ impl VmDriver {
}
validate_openshell_endpoint(&config.grpc_endpoint)?;
let _ = config.tls_paths()?;
initialize_additional_ca_snapshot(&mut config)?;
config.state_dir = absolute_state_dir(&config.state_dir)?;
#[cfg(target_os = "linux")]
@@ -935,6 +1353,24 @@ impl VmDriver {
Ok(destination)
}
/// Stage the captured destination trust into this sandbox's host-only
/// state directory. This intentionally happens per launch so stopped
/// sandboxes do not carry a stale source generation into a later host
/// supervisor process.
async fn stage_additional_ca_for_sandbox(
&self,
state_dir: &Path,
) -> Result<Option<PathBuf>, Status> {
let state_dir = state_dir.to_path_buf();
let snapshot = self.config.additional_ca_snapshot.clone();
tokio::task::spawn_blocking(move || {
stage_additional_ca_snapshot(&state_dir, snapshot.as_ref())
})
.await
.map_err(|error| Status::internal(format!("stage additional CA task panicked: {error}")))?
.map_err(|error| Status::internal(format!("stage host additional CA: {error}")))
}
async fn spawn_host_supervisor(
&self,
sandbox: &Sandbox,
@@ -944,6 +1380,15 @@ impl VmDriver {
auth_bundle: &openshell_core::jwt::SupervisorAuthBundle,
sandbox_owner: SandboxOwnerIdentity,
) -> Result<(Child, Option<fs::File>), Status> {
// The destination CA is staged on the host before either resolving
// the supervisor binary or spawning it. It never crosses into a VM
// disk or a guest environment.
let staged_network_trust = self.stage_additional_ca_for_sandbox(state_dir).await?;
let network_trust_args = additional_ca_supervisor_args(
self.config.additional_ca_snapshot.as_ref(),
staged_network_trust.as_deref(),
)
.map_err(Status::internal)?;
let supervisor_binary = self.host_supervisor_binary().await?;
let (openshell_endpoint, gateway_tls_server_name) =
host_control_openshell_endpoint(&self.config.grpc_endpoint)
@@ -1004,6 +1449,7 @@ impl VmDriver {
.arg("--workdir")
.arg("/sandbox")
.args(upstream_proxy_args)
.args(network_trust_args)
.env(
openshell_core::sandbox_env::ADMITTED_ISOLATION_BACKEND,
DRIVER_ADMITTED_BACKEND,
@@ -7872,8 +8318,9 @@ mod tests {
GpuResourceRequirements, ResourceRequirements,
};
use prost_types::{Struct, Value, value::Kind};
use rcgen::{BasicConstraints, CertificateParams, IsCa, KeyPair};
use std::fs;
use std::path::Path;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
use std::time::{SystemTime, UNIX_EPOCH};
use tonic::Code;
@@ -7994,6 +8441,283 @@ mod tests {
assert!(error.to_string().contains("openshell_endpoint"));
}
fn normalized_test_network_ca_bundle(certificate_count: usize) -> Vec<u8> {
let mut pem = String::new();
for _ in 0..certificate_count {
let mut params = CertificateParams::new(Vec::<String>::new()).unwrap();
params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained);
let key = KeyPair::generate().unwrap();
pem.push_str(&params.self_signed(&key).unwrap().pem());
}
pem.into_bytes()
}
fn write_gateway_network_ca_artifact(path: &Path, pem: &[u8]) {
let _ = fs::remove_file(path);
fs::write(path, pem).unwrap();
#[cfg(unix)]
fs::set_permissions(path, fs::Permissions::from_mode(0o444)).unwrap();
}
fn network_ca_config(path: PathBuf, pem: &[u8], certificate_count: usize) -> VmDriverConfig {
VmDriverConfig {
grpc_endpoint: "http://127.0.0.1:8080".to_string(),
additional_ca_bundle: Some(path),
additional_ca_digest: Some(format!("sha256:{:x}", Sha256::digest(pem))),
additional_ca_certificate_count: Some(certificate_count),
..Default::default()
}
}
#[test]
fn additional_ca_launch_options_are_all_or_none_and_snapshot_is_driver_owned() {
let path = PathBuf::from("/gateway/artifact/additional-ca.crt");
let digest = format!("sha256:{}", "a".repeat(64));
for config in [
VmDriverConfig {
additional_ca_bundle: Some(path.clone()),
..Default::default()
},
VmDriverConfig {
additional_ca_digest: Some(digest.clone()),
..Default::default()
},
VmDriverConfig {
additional_ca_certificate_count: Some(1),
..Default::default()
},
VmDriverConfig {
additional_ca_bundle: Some(path.clone()),
additional_ca_digest: Some(digest.clone()),
..Default::default()
},
VmDriverConfig {
additional_ca_bundle: Some(path),
additional_ca_certificate_count: Some(1),
..Default::default()
},
VmDriverConfig {
additional_ca_digest: Some(digest),
additional_ca_certificate_count: Some(1),
..Default::default()
},
] {
let mut config = config;
let error = initialize_additional_ca_snapshot(&mut config)
.expect_err("partial private launch metadata must be rejected");
assert!(error.contains("must be supplied together"), "{error}");
}
let mut no_trust = VmDriverConfig::default();
initialize_additional_ca_snapshot(&mut no_trust)
.expect("all absent is the explicit no-trust state");
assert!(no_trust.additional_ca_snapshot.is_none());
let supplied_snapshot = VmNetworkSupervisorTrustSnapshot {
bundle: NetworkSupervisorTrustBundle::new(
b"-----BEGIN CERTIFICATE-----\nprivate-fixture-bytes\n-----END CERTIFICATE-----\n"
.to_vec(),
1,
format!("sha256:{}", "b".repeat(64)),
PathBuf::from("/untrusted/caller-snapshot"),
),
};
let mut caller_snapshot = VmDriverConfig {
additional_ca_snapshot: Some(supplied_snapshot.clone()),
..Default::default()
};
let error = initialize_additional_ca_snapshot(&mut caller_snapshot)
.expect_err("callers must not preload driver-owned snapshot bytes");
assert!(error.contains("driver-owned"), "{error}");
let result = VmDriver::new(VmDriverConfig {
grpc_endpoint: "http://127.0.0.1:8080".to_string(),
bootstrap_image: "bootstrap:latest".to_string(),
additional_ca_snapshot: Some(supplied_snapshot),
..Default::default()
});
let error = futures::executor::block_on(result)
.err()
.expect("driver initialization must reject caller-provided snapshot state");
assert!(error.contains("driver-owned"), "{error}");
}
#[test]
fn additional_ca_snapshot_verifies_normalized_digest_and_certificate_count() {
let directory = tempfile::tempdir().unwrap();
let artifact = directory.path().join("gateway-additional-ca.crt");
let pem = normalized_test_network_ca_bundle(1);
write_gateway_network_ca_artifact(&artifact, &pem);
let mut accepted = network_ca_config(artifact.clone(), &pem, 1);
initialize_additional_ca_snapshot(&mut accepted)
.expect("gateway-normalized artifact should produce a snapshot");
let snapshot = accepted.additional_ca_snapshot.as_ref().expect("snapshot");
assert_eq!(snapshot.normalized_pem(), pem);
assert_eq!(snapshot.certificate_count(), 1);
assert!(accepted.additional_ca_bundle.is_none());
assert!(accepted.additional_ca_digest.is_none());
assert!(accepted.additional_ca_certificate_count.is_none());
let mut non_normalized_digest = network_ca_config(artifact.clone(), &pem, 1);
non_normalized_digest.additional_ca_digest = Some(format!("sha256:{}", "A".repeat(64)));
let error = initialize_additional_ca_snapshot(&mut non_normalized_digest)
.expect_err("digest must use normalized lowercase hexadecimal");
assert!(error.contains("normalized lowercase"), "{error}");
let tampered = b"private replacement CA bytes must not be disclosed\n";
fs::remove_file(&artifact).unwrap();
fs::write(&artifact, tampered).unwrap();
#[cfg(unix)]
fs::set_permissions(&artifact, fs::Permissions::from_mode(0o444)).unwrap();
let mut tampered_config = network_ca_config(artifact.clone(), &pem, 1);
let error = initialize_additional_ca_snapshot(&mut tampered_config)
.expect_err("a changed gateway artifact must fail closed");
assert!(error.contains("does not match expected digest"), "{error}");
assert!(!error.contains("private replacement"), "{error}");
write_gateway_network_ca_artifact(&artifact, &pem);
let mut wrong_count = network_ca_config(artifact, &pem, 2);
let error = initialize_additional_ca_snapshot(&mut wrong_count)
.expect_err("wrong certificate count must fail closed");
assert!(error.contains("certificate count 1, expected 2"), "{error}");
}
#[cfg(unix)]
#[test]
fn additional_ca_snapshot_read_is_bounded_and_never_follows_symlinks() {
use std::os::unix::fs::symlink;
let directory = tempfile::tempdir().unwrap();
let too_large = directory.path().join("too-large.crt");
let bytes = vec![b'x'; MAX_NETWORK_SUPERVISOR_TRUST_BUNDLE_BYTES + 1];
write_gateway_network_ca_artifact(&too_large, &bytes);
let mut oversized = network_ca_config(too_large, &bytes, 1);
let error = initialize_additional_ca_snapshot(&mut oversized)
.expect_err("oversized gateway artifact must be rejected before reading it");
assert!(error.contains("exceeds the"), "{error}");
let target = directory.path().join("target.crt");
let target_pem = normalized_test_network_ca_bundle(1);
write_gateway_network_ca_artifact(&target, &target_pem);
let link = directory.path().join("symlink.crt");
symlink(&target, &link).unwrap();
let mut symlinked = network_ca_config(link, &target_pem, 1);
let error = initialize_additional_ca_snapshot(&mut symlinked)
.expect_err("gateway artifact symlink must not be followed");
assert!(error.contains("not a regular file"), "{error}");
}
#[test]
fn additional_ca_staging_uses_immutable_snapshot_and_host_supervisor_argv() {
let directory = tempfile::tempdir().unwrap();
let artifact = directory.path().join("gateway-additional-ca.crt");
let pem = normalized_test_network_ca_bundle(1);
write_gateway_network_ca_artifact(&artifact, &pem);
let mut config = network_ca_config(artifact.clone(), &pem, 1);
initialize_additional_ca_snapshot(&mut config).unwrap();
let snapshot = config.additional_ca_snapshot.as_ref().unwrap();
// Change the original path after initialization. Staging must copy the
// retained Arc-backed bytes rather than reread this mutable source.
fs::remove_file(&artifact).unwrap();
fs::write(
&artifact,
b"private source replacement must not reach a launch\n",
)
.unwrap();
#[cfg(unix)]
fs::set_permissions(&artifact, fs::Permissions::from_mode(0o444)).unwrap();
let state_dir = directory.path().join("sandbox-state");
fs::create_dir(&state_dir).unwrap();
let staged = stage_additional_ca_snapshot(&state_dir, Some(snapshot))
.expect("stage captured bytes")
.expect("configured snapshot stages a host file");
assert_eq!(fs::read(&staged).unwrap(), pem);
#[cfg(unix)]
assert_eq!(
fs::metadata(&staged).unwrap().permissions().mode() & 0o777,
0o600
);
assert!(
!state_dir.join("overlay.ext4").exists(),
"destination trust must never be written to a guest overlay"
);
let args = additional_ca_supervisor_args(Some(snapshot), Some(&staged)).unwrap();
assert_eq!(
args,
["--additional-ca-bundle", staged.to_string_lossy().as_ref(),]
);
let diagnostic = format!("{config:?} {snapshot:?} {args:?}");
assert!(
!diagnostic.contains("private source replacement"),
"{diagnostic}"
);
assert!(
!diagnostic.contains("BEGIN CERTIFICATE"),
"PEM bytes must not appear in diagnostics or argv: {diagnostic}"
);
let guest_env = build_guest_environment(
&Sandbox {
id: "network-ca-sandbox".to_string(),
..Default::default()
},
&config,
);
assert!(
!guest_env
.iter()
.any(|entry| entry.contains("additional-ca")
|| entry.contains("BEGIN CERTIFICATE")),
"destination trust must not enter guest launch environment: {guest_env:?}"
);
}
#[test]
fn additional_ca_staging_clears_stale_host_material_for_no_trust() {
let directory = tempfile::tempdir().unwrap();
let artifact = directory.path().join("gateway-additional-ca.crt");
let pem = normalized_test_network_ca_bundle(1);
write_gateway_network_ca_artifact(&artifact, &pem);
let mut config = network_ca_config(artifact, &pem, 1);
initialize_additional_ca_snapshot(&mut config).unwrap();
let snapshot = config.additional_ca_snapshot.as_ref().unwrap();
let state_dir = directory.path().join("sandbox-state");
fs::create_dir(&state_dir).unwrap();
let staged = stage_additional_ca_snapshot(&state_dir, Some(snapshot))
.unwrap()
.unwrap();
assert!(staged.is_file());
let stale_temporary = state_dir.join(format!(
"{HOST_ADDITIONAL_CA_TEMP_PREFIX}stale{HOST_ADDITIONAL_CA_TEMP_SUFFIX}"
));
fs::write(&stale_temporary, b"stale trust bytes").unwrap();
assert_eq!(
stage_additional_ca_snapshot(&state_dir, None).unwrap(),
None
);
assert!(!staged.exists());
assert!(!stale_temporary.exists());
assert!(
additional_ca_supervisor_args(None, None)
.unwrap()
.is_empty()
);
// A subsequent configured generation recreates only the host file;
// no guest-overlay state participates in either transition.
let recreated = stage_additional_ca_snapshot(&state_dir, Some(snapshot))
.unwrap()
.unwrap();
assert_eq!(fs::read(recreated).unwrap(), pem);
assert!(!state_dir.join("overlay.ext4").exists());
}
struct TestTracing {
exporter: opentelemetry_sdk::trace::InMemorySpanExporter,
_provider: opentelemetry_sdk::trace::SdkTracerProvider,
crates/openshell-driver-vm/src/main.rs
+59
View File
@@ -142,6 +142,22 @@ struct Args {
#[arg(long = "guest-tls-key", env = "OPENSHELL_VM_TLS_KEY")]
guest_tls_key: Option<PathBuf>,
/// Gateway-owned normalized destination trust artifact. This hidden option
/// is supplied only by the in-process gateway VM factory.
#[arg(long, hide = true)]
additional_ca_bundle: Option<PathBuf>,
/// Expected SHA-256 generation of the gateway-owned destination trust
/// artifact. Supplied with the path so the child can authenticate its
/// one-time startup read without receiving CA bytes on argv or in env.
#[arg(long, hide = true)]
additional_ca_digest: Option<String>,
/// Expected certificate count in the gateway-normalized destination trust
/// artifact.
#[arg(long, hide = true)]
additional_ca_certificate_count: Option<usize>,
/// Corporate forward proxy for supervisor TLS egress.
#[arg(long, env = "OPENSHELL_VM_UPSTREAM_PROXY")]
upstream_proxy: Option<String>,
@@ -338,6 +354,10 @@ async fn main() -> Result<()> {
guest_tls_ca: args.guest_tls_ca.clone(),
guest_tls_cert: args.guest_tls_cert.clone(),
guest_tls_key: args.guest_tls_key.clone(),
additional_ca_bundle: args.additional_ca_bundle.clone(),
additional_ca_digest: args.additional_ca_digest.clone(),
additional_ca_certificate_count: args.additional_ca_certificate_count,
additional_ca_snapshot: None,
upstream_proxy: openshell_core::UpstreamProxyConfig {
https_proxy: args.upstream_proxy.clone(),
no_proxy: args.upstream_no_proxy.clone(),
@@ -827,6 +847,45 @@ mod tests {
assert!(args.upstream_proxy_ca_bundle.is_none());
}
#[test]
fn internal_network_trust_options_have_no_environment_aliases() {
let args = Args::parse_from([
"openshell-driver-vm",
"--additional-ca-bundle",
"/var/lib/openshell/network/additional-ca.crt",
"--additional-ca-digest",
"sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
"--additional-ca-certificate-count",
"2",
]);
assert_eq!(
args.additional_ca_bundle.as_deref(),
Some(PathBuf::from("/var/lib/openshell/network/additional-ca.crt").as_path())
);
assert_eq!(
args.additional_ca_digest.as_deref(),
Some("sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef")
);
assert_eq!(args.additional_ca_certificate_count, Some(2));
temp_env::with_vars(
[
(
"OPENSHELL_VM_ADDITIONAL_CA_BUNDLE",
Some("/tmp/untrusted.crt"),
),
("OPENSHELL_VM_ADDITIONAL_CA_DIGEST", Some("sha256:bad")),
("OPENSHELL_VM_ADDITIONAL_CA_CERTIFICATE_COUNT", Some("999")),
],
|| {
let args = Args::parse_from(["openshell-driver-vm"]);
assert!(args.additional_ca_bundle.is_none());
assert!(args.additional_ca_digest.is_none());
assert!(args.additional_ca_certificate_count.is_none());
},
);
}
#[test]
fn peer_authorization_accepts_matching_uid_and_pid() {
authorize_peer_credentials(
crates/openshell-gateway/Cargo.toml
+2
View File
@@ -79,5 +79,7 @@ vendored-z3 = ["openshell-server/vendored-z3"]
workspace = true
[dev-dependencies]
openshell-crypto = { workspace = true, default-features = false, features = ["aws-lc"] }
rcgen = { workspace = true }
tempfile = "3"
toml = { workspace = true }
crates/openshell-gateway/src/lib.rs
+50 -4
View File
@@ -265,6 +265,10 @@ impl openshell_server::ComputeDriverFactory for KubernetesFactory {
true
}
fn supports_network_supervisor_trust(&self) -> bool {
true
}
fn validate_config(
&self,
context: openshell_server::ComputeDriverConfigContext<'_>,
@@ -287,8 +291,19 @@ impl openshell_server::ComputeDriverFactory for KubernetesFactory {
config.supervisor_gateway_name = Some(context.gateway_name().to_string());
// With the gateway supervisor topology, the gateway owns the shared
// supervisor's assignments and token; the driver adapts them to
// Kubernetes objects.
// Kubernetes objects. The experimental shared supervisor topology
// does not support the network supervisor additional destination CA;
// refuse the combination fail-closed rather than silently
// supervising generations without the operator's trust bundle.
let network_trust = context.network_trust_bundle().cloned();
let driver = if config.shared_supervisor.is_some() {
if network_trust.is_some() {
return Err(openshell_core::Error::execution(
"the experimental shared supervisor topology does not support the \
network supervisor additional destination CA"
.to_string(),
));
}
openshell_driver_kubernetes::KubernetesComputeDriver::with_shared_supervision(
config,
context.shutdown_receiver(),
@@ -299,6 +314,7 @@ impl openshell_server::ComputeDriverFactory for KubernetesFactory {
openshell_driver_kubernetes::KubernetesComputeDriver::new(
config,
context.shutdown_receiver(),
network_trust,
)
.await
}
@@ -344,6 +360,10 @@ impl openshell_server::ComputeDriverFactory for DockerFactory {
true
}
fn supports_network_supervisor_trust(&self) -> bool {
true
}
fn validate_config(
&self,
context: openshell_server::ComputeDriverConfigContext<'_>,
@@ -369,6 +389,7 @@ impl openshell_server::ComputeDriverFactory for DockerFactory {
context.gateway_bind_address(),
context.gateway_log_level(),
&config,
context.network_trust_bundle().cloned(),
)
.await
.map_err(|error| openshell_core::Error::execution(error.to_string()))?;
@@ -410,6 +431,10 @@ impl openshell_server::ComputeDriverFactory for PodmanFactory {
true
}
fn supports_network_supervisor_trust(&self) -> bool {
true
}
fn validate_config(
&self,
context: openshell_server::ComputeDriverConfigContext<'_>,
@@ -432,9 +457,12 @@ impl openshell_server::ComputeDriverFactory for PodmanFactory {
config.supervisor_gateway_name = Some(context.gateway_name().to_string());
require_guest_tls_for_local_driver(&context, "podman")?;
apply_guest_tls(&mut config.guest_tls_ca, context.guest_tls_ca());
let driver = openshell_driver_podman::PodmanComputeDriver::new(config)
.await
.map_err(|error| openshell_core::Error::execution(error.to_string()))?;
let driver = openshell_driver_podman::PodmanComputeDriver::new(
config,
context.network_trust_bundle().cloned(),
)
.await
.map_err(|error| openshell_core::Error::execution(error.to_string()))?;
let driver = openshell_driver_podman::ComputeDriverService::new_in_process(driver);
Ok(openshell_server::ComputeDriverInstance::InProcess(
std::sync::Arc::new(driver),
@@ -482,6 +510,10 @@ impl openshell_server::ComputeDriverFactory for VmFactory {
.map_err(openshell_core::Error::config)
}
fn supports_network_supervisor_trust(&self) -> bool {
true
}
fn validate_config(
&self,
context: openshell_server::ComputeDriverConfigContext<'_>,
@@ -520,6 +552,7 @@ impl openshell_server::ComputeDriverFactory for VmFactory {
context.gateway_name(),
&config,
context.otlp_config(),
context.network_trust_bundle(),
)
.await?;
Ok(openshell_server::ComputeDriverInstance::ManagedRemote(
@@ -704,4 +737,17 @@ mod tests {
expected
);
}
#[cfg(not(target_os = "windows"))]
#[test]
fn first_party_network_supervisor_drivers_opt_into_trust_propagation() {
#[cfg(feature = "compute-driver-docker")]
assert!(<DockerFactory as openshell_server::ComputeDriverFactory>::supports_network_supervisor_trust(&DockerFactory));
#[cfg(feature = "compute-driver-kubernetes")]
assert!(<KubernetesFactory as openshell_server::ComputeDriverFactory>::supports_network_supervisor_trust(&KubernetesFactory));
#[cfg(feature = "compute-driver-podman")]
assert!(<PodmanFactory as openshell_server::ComputeDriverFactory>::supports_network_supervisor_trust(&PodmanFactory));
#[cfg(feature = "compute-driver-vm")]
assert!(<VmFactory as openshell_server::ComputeDriverFactory>::supports_network_supervisor_trust(&VmFactory));
}
}
crates/openshell-gateway/src/vm.rs
+64 -3
View File
@@ -31,7 +31,7 @@ use hyper_util::rt::TokioIo;
use openshell_core::proto::compute::v1::{
GetCapabilitiesRequest, compute_driver_client::ComputeDriverClient,
};
use openshell_core::{Error, Result, UpstreamProxyConfig};
use openshell_core::{Error, NetworkSupervisorTrustBundle, Result, UpstreamProxyConfig};
#[cfg(unix)]
use openshell_otel::TraceContextInterceptor;
use openshell_server::AcquiredRemoteDriverEndpoint;
@@ -531,6 +531,7 @@ pub async fn spawn(
gateway_name: &str,
vm_config: &VmComputeConfig,
otlp_config: Option<&OtlpConfig>,
additional_ca_bundle: Option<&NetworkSupervisorTrustBundle>,
) -> Result<AcquiredRemoteDriverEndpoint> {
vm_config.validate_configuration()?;
let driver_bin = resolve_compute_driver_bin(vm_config)?;
@@ -580,6 +581,7 @@ pub async fn spawn(
command.arg("--guest-tls-ca").arg(tls.ca);
}
append_vm_proxy_and_spiffe_args(&mut command, vm_config);
append_vm_network_trust_args(&mut command, additional_ca_bundle);
if vm_config.enable_bind_mounts {
command.arg("--enable-bind-mounts");
}
@@ -638,6 +640,25 @@ fn append_vm_rootfs_tar_args(command: &mut Command, config: &VmComputeConfig) {
}
}
#[cfg(unix)]
fn append_vm_network_trust_args(
command: &mut Command,
bundle: Option<&NetworkSupervisorTrustBundle>,
) {
if let Some(bundle) = bundle {
// Only stable, non-secret artifact metadata crosses the process
// boundary. The child reads and verifies the bounded artifact once,
// then retains its own immutable snapshot for sandbox launches.
command
.arg("--additional-ca-bundle")
.arg(bundle.artifact_path());
command.arg("--additional-ca-digest").arg(bundle.digest());
command
.arg("--additional-ca-certificate-count")
.arg(bundle.certificate_count().to_string());
}
}
#[cfg(unix)]
fn append_vm_proxy_and_spiffe_args(command: &mut Command, config: &VmComputeConfig) {
let proxy = &config.upstream_proxy;
@@ -694,6 +715,7 @@ pub async fn spawn(
_gateway_name: &str,
_vm_config: &VmComputeConfig,
_otlp_config: Option<&OtlpConfig>,
_additional_ca_bundle: Option<&NetworkSupervisorTrustBundle>,
) -> Result<AcquiredRemoteDriverEndpoint> {
Err(Error::config(
"the vm compute driver requires unix domain socket support",
@@ -775,13 +797,13 @@ async fn connect_compute_driver(socket_path: &Path) -> Result<Channel> {
#[cfg(all(test, unix))]
mod tests {
use super::{
VmComputeConfig, append_otlp_args, append_vm_identity_args,
VmComputeConfig, append_otlp_args, append_vm_identity_args, append_vm_network_trust_args,
append_vm_proxy_and_spiffe_args, append_vm_rootfs_tar_args, compute_driver_guest_tls_paths,
compute_driver_socket_path, current_euid, prepare_compute_driver_socket_path,
prepare_vm_state_dir, resolve_compute_driver_bin, resolve_driver_search_dirs,
validate_vm_sandbox_identity,
};
use openshell_core::UpstreamProxyConfig;
use openshell_core::{NetworkSupervisorTrustBundle, UpstreamProxyConfig};
use openshell_server::config_file::{
OcsfOtlpConfig, OcsfSchemaVersion, OtlpConfig, OtlpTracesConfig,
};
@@ -943,6 +965,45 @@ mod tests {
assert_eq!(command.as_std().get_args().count(), 0);
}
#[test]
fn vm_driver_command_forwards_only_network_trust_artifact_metadata() {
let mut command = tokio::process::Command::new("openshell-driver-vm");
let pem =
b"-----BEGIN CERTIFICATE-----\nprivate-fixture-bytes\n-----END CERTIFICATE-----\n";
let bundle = NetworkSupervisorTrustBundle::new(
pem.to_vec(),
2,
"sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
PathBuf::from("/var/lib/openshell/network/additional-ca.crt"),
);
append_vm_network_trust_args(&mut command, Some(&bundle));
let args = command
.as_std()
.get_args()
.map(|arg| arg.to_string_lossy().into_owned())
.collect::<Vec<_>>();
assert_eq!(
args,
[
"--additional-ca-bundle",
"/var/lib/openshell/network/additional-ca.crt",
"--additional-ca-digest",
"sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
"--additional-ca-certificate-count",
"2",
]
);
assert!(!args.iter().any(|arg| arg == "OPENSHELL_TLS_CA"));
assert!(
!args.iter().any(|arg| arg.contains("private-fixture-bytes")),
"CA material must not cross the process boundary in argv: {args:?}"
);
let mut unset = tokio::process::Command::new("openshell-driver-vm");
append_vm_network_trust_args(&mut unset, None);
assert_eq!(unset.as_std().get_args().count(), 0);
}
#[test]
fn invalid_corporate_proxy_config_is_rejected_before_the_driver_starts() {
let err = UpstreamProxyConfig {
crates/openshell-gateway/tests/network_trust_startup.rs
+238
View File
@@ -0,0 +1,238 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
//! Process-level startup coverage for global network-supervisor trust material.
#![cfg(unix)]
use std::fs;
use std::path::{Path, PathBuf};
use std::process::{Command, Output};
use openshell_crypto::pki::generate_simple_self_signed;
use tempfile::TempDir;
const CONFIG_FIELD: &str = "openshell.supervisor.network.additional_ca_bundle";
struct GatewayFixture {
root: TempDir,
config_path: PathBuf,
}
impl GatewayFixture {
fn new(source_path: &Path) -> Self {
let root = tempfile::tempdir().expect("create gateway fixture directory");
let config_path = root.path().join("gateway.toml");
fs::write(
&config_path,
format!(
"[openshell]\nversion = 2\n\n[openshell.supervisor.network]\nadditional_ca_bundle = \"{}\"\n",
source_path.display()
),
)
.expect("write gateway config");
Self { root, config_path }
}
fn run(&self, driver: &str, socket: Option<&Path>) -> Output {
let mut command = Command::new(env!("CARGO_BIN_EXE_openshell-gateway"));
command
.args([
"--config",
self.config_path.to_str().expect("UTF-8 config path"),
"--db-url",
"sqlite::memory:",
"--disable-tls",
"--compute-driver",
driver,
"--log-level",
"info",
])
.env("XDG_STATE_HOME", self.root.path().join("state"))
.env("XDG_CONFIG_HOME", self.root.path().join("config"))
.env_remove("RUST_LOG")
.env_remove("OPENSHELL_DRIVERS")
.env_remove("OPENSHELL_COMPUTE_DRIVER")
.env_remove("OPENSHELL_COMPUTE_DRIVER_SOCKET")
.env_remove("OPENSHELL_GATEWAY_CONFIG");
if let Some(socket) = socket {
command.args([
"--compute-driver-socket",
socket.to_str().expect("UTF-8 socket path"),
]);
}
command.output().expect("run openshell-gateway")
}
}
fn combined_output(output: &Output) -> String {
let raw = format!(
"{}{}",
String::from_utf8_lossy(&output.stdout),
String::from_utf8_lossy(&output.stderr)
);
let raw = raw.as_bytes();
let mut plain = Vec::with_capacity(raw.len());
let mut index = 0;
while index < raw.len() {
if raw[index] == 0x1b {
index += 1;
while index < raw.len() {
let byte = raw[index];
index += 1;
if byte.is_ascii_alphabetic() {
break;
}
}
} else {
plain.push(raw[index]);
index += 1;
}
}
String::from_utf8_lossy(&plain)
.replace('│', " ")
.split_whitespace()
.collect::<Vec<_>>()
.join(" ")
}
fn pem_payload_marker(pem: &str) -> &str {
pem.lines()
.find(|line| line.len() > 32 && !line.starts_with("-----"))
.expect("PEM fixture payload line")
}
/// Substring check that ignores the gateway CLI's diagnostic line wrapping.
///
/// The rendered diagnostic wraps at a fixed width with `│` continuation
/// markers, and long tokens (for example source paths) are hard-broken
/// mid-word. Squashing whitespace and continuation glyphs on both sides
/// keeps assertions independent of where the wrap lands.
fn contains_wrapped(haystack: &str, needle: &str) -> bool {
fn squash(text: &str) -> String {
text.chars()
.filter(|c| !c.is_whitespace() && *c != '│')
.collect()
}
squash(haystack).contains(&squash(needle))
}
fn generated_certificate() -> openshell_crypto::pki::CertifiedKey {
generate_simple_self_signed(vec!["destination.example".to_string()])
.expect("generate certificate fixture")
}
#[test]
fn valid_bundle_reports_only_redacted_metadata_before_custom_driver_rejection() {
let source_dir = tempfile::tempdir().expect("create source directory");
let certificate = generated_certificate();
let source_path = source_dir.path().join("destination-ca.pem");
let pem = certificate.cert.pem();
fs::write(&source_path, &pem).expect("write certificate fixture");
let fixture = GatewayFixture::new(&source_path);
let output = fixture.run("custom-driver", None);
let diagnostic = combined_output(&output);
assert!(!output.status.success(), "custom driver must be rejected");
assert!(
contains_wrapped(
&diagnostic,
"Network supervisor additional destination trust configured"
),
"missing startup metadata in:\n{diagnostic}"
);
assert!(
contains_wrapped(&diagnostic, "certificate_count=1"),
"{diagnostic}"
);
assert!(contains_wrapped(&diagnostic, "sha256:"), "{diagnostic}");
assert!(contains_wrapped(&diagnostic, CONFIG_FIELD), "{diagnostic}");
assert!(
contains_wrapped(&diagnostic, "custom-driver"),
"{diagnostic}"
);
assert!(contains_wrapped(&diagnostic, "unsupported"), "{diagnostic}");
assert!(
!contains_wrapped(&diagnostic, pem_payload_marker(&pem)),
"certificate payload leaked"
);
}
#[test]
fn invalid_bundles_fail_startup_with_actionable_redacted_errors() {
let source_dir = tempfile::tempdir().expect("create source directory");
let generated = generated_certificate();
let private_key = generated.key_pair.serialize_pem().unwrap();
let cases = [
(
source_dir.path().join("missing.pem"),
None,
"could not be read",
),
(
source_dir.path().join("empty.pem"),
Some(String::new()),
"contains no PEM certificate",
),
(
source_dir.path().join("private-key.pem"),
Some(private_key.clone()),
"contains a non-certificate PEM",
),
];
for (source_path, contents, expected) in cases {
if let Some(contents) = contents {
fs::write(&source_path, contents).expect("write invalid fixture");
}
let fixture = GatewayFixture::new(&source_path);
let output = fixture.run("docker", None);
let diagnostic = combined_output(&output);
assert!(!output.status.success(), "invalid bundle must fail startup");
assert!(contains_wrapped(&diagnostic, CONFIG_FIELD), "{diagnostic}");
assert!(
contains_wrapped(&diagnostic, &source_path.display().to_string()),
"{diagnostic}"
);
assert!(contains_wrapped(&diagnostic, expected), "{diagnostic}");
assert!(
!contains_wrapped(&diagnostic, pem_payload_marker(&private_key)),
"private key payload leaked"
);
assert!(
!contains_wrapped(&diagnostic, "Starting OpenShell server"),
"driver/server startup began after invalid material:\n{diagnostic}"
);
}
}
#[test]
fn configured_remote_endpoint_is_rejected_before_connection() {
let source_dir = tempfile::tempdir().expect("create source directory");
let certificate = generated_certificate();
let source_path = source_dir.path().join("destination-ca.pem");
fs::write(&source_path, certificate.cert.pem()).expect("write certificate fixture");
let fixture = GatewayFixture::new(&source_path);
let socket_path = fixture.root.path().join("must-not-connect.sock");
let output = fixture.run("docker", Some(&socket_path));
let diagnostic = combined_output(&output);
assert!(!output.status.success(), "remote override must be rejected");
assert!(contains_wrapped(&diagnostic, CONFIG_FIELD), "{diagnostic}");
assert!(
contains_wrapped(&diagnostic, "unsupported remote"),
"{diagnostic}"
);
assert!(contains_wrapped(&diagnostic, "endpoint"), "{diagnostic}");
assert!(
!contains_wrapped(&diagnostic, "failed to create compute runtime"),
"gateway attempted remote-driver construction:\n{diagnostic}"
);
assert!(
!contains_wrapped(&diagnostic, "No such file or directory"),
"gateway attempted to connect to the sentinel socket:\n{diagnostic}"
);
}
crates/openshell-sandbox/src/child_env.rs
+46
View File
@@ -3,6 +3,42 @@
use std::path::Path;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum TlsEnvironmentMode {
/// `OpenShell`'s generated trust files are authoritative for mediated traffic.
Override,
/// Preserve caller-provided TLS variables and fill only missing values.
FillMissing,
}
const TLS_ENVIRONMENT_VARIABLES: [&str; 6] = [
"NODE_EXTRA_CA_CERTS",
"DENO_CERT",
"SSL_CERT_FILE",
"REQUESTS_CA_BUNDLE",
"CURL_CA_BUNDLE",
"GIT_SSL_CAINFO",
];
#[must_use]
pub fn is_tls_environment_variable(key: &str) -> bool {
TLS_ENVIRONMENT_VARIABLES.contains(&key)
}
pub fn tls_env_vars_for_mode<S: std::hash::BuildHasher>(
ca_cert_path: &Path,
combined_bundle_path: &Path,
mode: TlsEnvironmentMode,
user_environment: &std::collections::HashMap<String, String, S>,
) -> Vec<(&'static str, String)> {
tls_env_vars(ca_cert_path, combined_bundle_path)
.into_iter()
.filter(|(key, _)| {
matches!(mode, TlsEnvironmentMode::Override) || !user_environment.contains_key(*key)
})
.collect()
}
pub fn tls_env_vars(
ca_cert_path: &Path,
combined_bundle_path: &Path,
@@ -49,5 +85,15 @@ mod tests {
assert!(stdout.contains("REQUESTS_CA_BUNDLE=/etc/openshell-tls/ca-bundle.pem"));
assert!(stdout.contains("CURL_CA_BUNDLE=/etc/openshell-tls/ca-bundle.pem"));
assert!(stdout.contains("GIT_SSL_CAINFO=/etc/openshell-tls/ca-bundle.pem"));
let keys = tls_env_vars(ca_cert_path, combined_bundle_path).map(|(key, _)| key);
assert!(
!keys.contains(&openshell_core::sandbox_env::TLS_CA),
"destination/child trust must not overwrite gateway mTLS trust"
);
assert_ne!(
ca_cert_path,
Path::new(openshell_core::container_paths::TLS_CA_MOUNT_PATH)
);
}
}
crates/openshell-sandbox/src/delegated.rs
+5
View File
@@ -86,6 +86,9 @@ pub async fn spawn_workload(
launcher.clone(),
));
// The delegated workload runs behind OpenShell's mediated network path,
// so the generated trust files remain authoritative for its TLS settings.
let tls_environment_mode = crate::child_env::TlsEnvironmentMode::Override;
#[cfg(target_os = "linux")]
let mut handle = ProcessHandle::spawn(
launcher,
@@ -95,6 +98,7 @@ pub async fn spawn_workload(
interactive,
policy,
ca_file_paths.as_ref(),
tls_environment_mode,
&provider_env,
)
.wrap_err("spawn delegated workload process")?;
@@ -106,6 +110,7 @@ pub async fn spawn_workload(
interactive,
policy,
ca_file_paths.as_ref(),
tls_environment_mode,
&provider_env,
)?;
crates/openshell-sandbox/src/process.rs
+52 -6
View File
@@ -428,6 +428,7 @@ impl ProcessHandle {
interactive: bool,
policy: &SandboxPolicy,
ca_paths: Option<&(PathBuf, PathBuf)>,
tls_environment_mode: child_env::TlsEnvironmentMode,
provider_env: &HashMap<String, String>,
) -> Result<Self> {
Self::spawn_impl(
@@ -438,6 +439,7 @@ impl ProcessHandle {
interactive,
policy,
ca_paths,
tls_environment_mode,
provider_env,
)
}
@@ -456,6 +458,7 @@ impl ProcessHandle {
interactive: bool,
policy: &SandboxPolicy,
ca_paths: Option<&(PathBuf, PathBuf)>,
tls_environment_mode: child_env::TlsEnvironmentMode,
provider_env: &HashMap<String, String>,
) -> Result<Self> {
Self::spawn_impl(
@@ -465,6 +468,7 @@ impl ProcessHandle {
interactive,
policy,
ca_paths,
tls_environment_mode,
provider_env,
)
}
@@ -479,6 +483,7 @@ impl ProcessHandle {
interactive: bool,
policy: &SandboxPolicy,
ca_paths: Option<&(PathBuf, PathBuf)>,
tls_environment_mode: child_env::TlsEnvironmentMode,
provider_env: &HashMap<String, String>,
) -> Result<Self> {
let mut cmd = Command::new(program);
@@ -511,12 +516,13 @@ impl ProcessHandle {
// inherited environment. The entrypoint drops to the sandbox user
// before `exec`; without this strip, sandbox code could recover
// supervisor credentials from its inherited environment.
let user_environment = configured_user_environment();
apply_canonical_process_environment(
&mut cmd,
policy,
workspace,
interactive,
&configured_user_environment(),
&user_environment,
);
strip_supervisor_only_env(&mut cmd);
@@ -528,9 +534,28 @@ impl ProcessHandle {
strip_proxy_env(&mut cmd);
// Set TLS trust store env vars so sandbox processes trust the ephemeral CA
// In interception mode OpenShell trust is authoritative. In direct
// additional-CA mode, retain all caller values and only fill gaps.
if let Some((ca_cert_path, combined_bundle_path)) = ca_paths {
for (key, value) in child_env::tls_env_vars(ca_cert_path, combined_bundle_path) {
// Provider credentials are injected before this point. Reassert
// caller TLS values in additive mode so a provider environment
// cannot accidentally replace a caller-selected trust store.
if matches!(
tls_environment_mode,
child_env::TlsEnvironmentMode::FillMissing
) {
for (key, value) in &user_environment {
if child_env::is_tls_environment_variable(key) {
cmd.env(key, value);
}
}
}
for (key, value) in child_env::tls_env_vars_for_mode(
ca_cert_path,
combined_bundle_path,
tls_environment_mode,
&user_environment,
) {
cmd.env(key, value);
}
}
@@ -639,6 +664,7 @@ impl ProcessHandle {
interactive: bool,
policy: &SandboxPolicy,
ca_paths: Option<&(PathBuf, PathBuf)>,
tls_environment_mode: child_env::TlsEnvironmentMode,
provider_env: &HashMap<String, String>,
) -> Result<Self> {
let mut cmd = Command::new(program);
@@ -671,12 +697,13 @@ impl ProcessHandle {
// Strip supervisor-only identity material from the entrypoint's
// inherited environment.
let user_environment = configured_user_environment();
apply_canonical_process_environment(
&mut cmd,
policy,
workspace,
interactive,
&configured_user_environment(),
&user_environment,
);
strip_supervisor_only_env(&mut cmd);
@@ -688,9 +715,28 @@ impl ProcessHandle {
strip_proxy_env(&mut cmd);
// Set TLS trust store env vars so sandbox processes trust the ephemeral CA
// In interception mode OpenShell trust is authoritative. In direct
// additional-CA mode, retain all caller values and only fill gaps.
if let Some((ca_cert_path, combined_bundle_path)) = ca_paths {
for (key, value) in child_env::tls_env_vars(ca_cert_path, combined_bundle_path) {
// Provider credentials are injected before this point. Reassert
// caller TLS values in additive mode so a provider environment
// cannot accidentally replace a caller-selected trust store.
if matches!(
tls_environment_mode,
child_env::TlsEnvironmentMode::FillMissing
) {
for (key, value) in &user_environment {
if child_env::is_tls_environment_variable(key) {
cmd.env(key, value);
}
}
}
for (key, value) in child_env::tls_env_vars_for_mode(
ca_cert_path,
combined_bundle_path,
tls_environment_mode,
&user_environment,
) {
cmd.env(key, value);
}
}
crates/openshell-server/src/cli.rs
+253
View File
@@ -401,6 +401,15 @@ fn prepare_server_config_with_drivers(
} else {
None
};
// Normalize sources now, but defer artifact staging until the selected
// driver has been proven able to propagate the bundle. This keeps a
// rejected startup/preflight from leaving gateway state behind.
let normalized_network_trust = file
.as_ref()
.map(|file| crate::network_trust::normalize_from_config(&file.openshell.supervisor.network))
.transpose()
.map_err(|error| miette::miette!("{error}"))?
.flatten();
if let Some(file) = file.as_ref() {
merge_file_into_args(args, &file.openshell.gateway, matches);
}
@@ -665,10 +674,46 @@ fn prepare_server_config_with_drivers(
// user-authored config file.
config.gateway_jwt = gateway_jwt;
if let Some(normalized) = normalized_network_trust.as_ref() {
// Use the same resolver as runtime construction, with the explicit
// configuration marker but no synthetic/staged artifact. This rejects
// remote endpoints and factories that have not opted into trust
// propagation before any state is written.
let driver_startup = crate::compute::driver_config::DriverStartupContext {
file: file.as_ref(),
guest_tls: guest_tls.as_ref(),
network_trust: None,
network_trust_configured: true,
gateway_port: config.bind_address.port(),
gateway_tls_enabled: config.tls.is_some(),
endpoint_overrides: &config.compute_driver_endpoints,
};
crate::validate_compute_driver_config(
compute_drivers,
compute_driver.name(),
&config.name,
config.bind_address,
&config.log_level,
driver_startup,
false,
)
.map_err(|error| {
miette::miette!(
"Network supervisor additional destination trust configured: certificate_count={} digest={}: {error}",
normalized.certificate_count(),
normalized.digest()
)
})?;
}
let network_trust = normalized_network_trust
.map(crate::network_trust::stage_normalized)
.transpose()?;
Ok(ServerStartupConfig {
config,
config_file: file,
guest_tls,
network_trust,
compute_driver,
legacy_compute_driver_env_seen,
})
@@ -748,6 +793,13 @@ async fn run_from_args(
if prepared.legacy_compute_driver_env_seen {
warn!("OPENSHELL_DRIVERS is deprecated; migrate to OPENSHELL_COMPUTE_DRIVER");
}
if let Some(bundle) = prepared.network_trust.as_ref() {
info!(
certificate_count = bundle.certificate_count(),
digest = bundle.digest(),
"Network supervisor additional destination trust configured"
);
}
let has_client_ca = prepared
.config
@@ -833,6 +885,10 @@ impl crate::ComputeDriverFactory for PreflightTestFactory {
true
}
fn supports_network_supervisor_trust(&self) -> bool {
true
}
async fn build(
&self,
_context: crate::ComputeDriverBuildContext<'_>,
@@ -932,6 +988,12 @@ fn run_effective_config_preflight(
let empty_file = ConfigFile::default();
let semantic_file = file.as_ref().unwrap_or(&empty_file);
validate_preflight_semantics(&run, matches, semantic_file)?;
// Preflight must exercise the same full source parsing as startup,
// while deliberately leaving state artifacts untouched.
let network_trust_configured = crate::network_trust::normalize_from_config(
&semantic_file.openshell.supervisor.network,
)?
.is_some();
let mut endpoint_overrides = BTreeMap::new();
if let Some(selection) = selection.as_ref()
@@ -942,6 +1004,8 @@ fn run_effective_config_preflight(
let driver_startup = crate::compute::driver_config::DriverStartupContext {
file: file.as_ref(),
guest_tls: None,
network_trust: None,
network_trust_configured,
gateway_port: run.port,
gateway_tls_enabled: !run.disable_tls,
endpoint_overrides: &endpoint_overrides,
@@ -1539,6 +1603,43 @@ mod tests {
}
}
static ENV_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());
struct EnvVarGuard {
key: &'static str,
original: Option<String>,
}
impl EnvVarGuard {
#[allow(unsafe_code)]
fn set(key: &'static str, value: &str) -> Self {
let original = std::env::var(key).ok();
// SAFETY: tests serialize environment mutation with ENV_LOCK.
unsafe { std::env::set_var(key, value) };
Self { key, original }
}
#[allow(unsafe_code)]
fn remove(key: &'static str) -> Self {
let original = std::env::var(key).ok();
// SAFETY: tests serialize environment mutation with ENV_LOCK.
unsafe { std::env::remove_var(key) };
Self { key, original }
}
}
impl Drop for EnvVarGuard {
#[allow(unsafe_code)]
fn drop(&mut self) {
match self.original.as_deref() {
// SAFETY: tests serialize environment mutation with ENV_LOCK.
Some(value) => unsafe { std::env::set_var(self.key, value) },
// SAFETY: tests serialize environment mutation with ENV_LOCK.
None => unsafe { std::env::remove_var(self.key) },
}
}
}
static REGISTRY_DETECTION_CALLS: AtomicUsize = AtomicUsize::new(0);
static REJECTING_VALIDATION_CALLS: AtomicUsize = AtomicUsize::new(0);
@@ -1563,6 +1664,10 @@ mod tests {
true
}
fn supports_network_supervisor_trust(&self) -> bool {
true
}
async fn build(
&self,
_context: crate::ComputeDriverBuildContext<'_>,
@@ -2546,6 +2651,53 @@ mod tests {
});
}
#[test]
fn config_preflight_normalizes_network_trust_without_creating_an_artifact() {
let _lock = ENV_LOCK
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let config_home = tempfile::tempdir().unwrap();
let source_home = tempfile::tempdir().unwrap();
let state_parent = tempfile::tempdir().unwrap();
let state_home = state_parent.path().join("not-created");
let source = source_home.path().join("destination-ca.pem");
let certificate = rcgen::generate_simple_self_signed(vec!["destination.example".into()])
.expect("test certificate");
std::fs::write(&source, certificate.cert.pem()).unwrap();
let config = config_home.path().join("gateway.toml");
std::fs::write(
&config,
format!(
"[openshell]\nversion = 2\n\n[openshell.supervisor.network]\nadditional_ca_bundle = \"{}\"\n",
source.display()
),
)
.unwrap();
let _config_env = EnvVarGuard::remove("OPENSHELL_GATEWAY_CONFIG");
let _state = EnvVarGuard::set("XDG_STATE_HOME", state_home.to_str().unwrap());
let (run, matches) = parse_with_args(&[
"openshell-gateway",
"--compute-driver",
"test",
"--disable-tls",
]);
super::run_config_preflight(
super::ConfigPreflightArgs {
path: Some(config),
..Default::default()
},
run,
&matches,
)
.expect("valid trust source and capable selected driver pass preflight");
assert!(
!state_home.exists(),
"preflight must not create a normalized trust artifact"
);
}
#[test]
fn config_preflight_explicit_path_overrides_environment_selection() {
let dir = tempfile::tempdir().unwrap();
@@ -3017,6 +3169,107 @@ mod tests {
});
}
#[test]
fn prepare_server_config_normalizes_global_network_trust_before_driver_startup() {
let _lock = ENV_LOCK
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let state = tempfile::tempdir().unwrap();
let source_dir = tempfile::tempdir().unwrap();
let config_dir = tempfile::tempdir().unwrap();
let _state = EnvVarGuard::set("XDG_STATE_HOME", state.path().to_str().unwrap());
let _config = EnvVarGuard::set("XDG_CONFIG_HOME", config_dir.path().to_str().unwrap());
let certificate = rcgen::generate_simple_self_signed(vec!["destination.example".into()])
.expect("test certificate");
let source = source_dir.path().join("destination-ca.pem");
std::fs::write(&source, certificate.cert.pem()).unwrap();
let config_path = config_dir.path().join("gateway.toml");
std::fs::write(
&config_path,
format!(
"[openshell]\nversion = 2\n\n[openshell.supervisor.network]\nadditional_ca_bundle = \"{}\"\n",
source.display()
),
)
.unwrap();
let (mut args, matches) = parse_with_args(&[
"openshell-gateway",
"--config",
config_path.to_str().unwrap(),
"--db-url",
"sqlite::memory:",
"--disable-tls",
"--compute-driver",
"local",
]);
let prepared = super::prepare_server_config_with_drivers(
&mut args,
&matches,
&detected_local_registry(),
)
.expect("global network trust should normalize during startup preparation");
let bundle = prepared.network_trust.expect("normalized trust bundle");
assert_eq!(bundle.certificate_count(), 1);
assert!(bundle.artifact_path().is_file());
assert_eq!(std::fs::read(bundle.artifact_path()).unwrap(), bundle.pem());
}
#[test]
fn prepare_server_config_rejects_network_trust_endpoint_override_without_artifact() {
let _lock = ENV_LOCK
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let state = tempfile::tempdir().unwrap();
let source_dir = tempfile::tempdir().unwrap();
let config_dir = tempfile::tempdir().unwrap();
let _state = EnvVarGuard::set("XDG_STATE_HOME", state.path().to_str().unwrap());
let _config = EnvVarGuard::set("XDG_CONFIG_HOME", config_dir.path().to_str().unwrap());
let certificate = rcgen::generate_simple_self_signed(vec!["destination.example".into()])
.expect("test certificate");
let source = source_dir.path().join("destination-ca.pem");
std::fs::write(&source, certificate.cert.pem()).unwrap();
let config_path = config_dir.path().join("gateway.toml");
std::fs::write(
&config_path,
format!(
"[openshell]\nversion = 2\n\n[openshell.supervisor.network]\nadditional_ca_bundle = \"{}\"\n",
source.display()
),
)
.unwrap();
let (mut args, matches) = parse_with_args(&[
"openshell-gateway",
"--config",
config_path.to_str().unwrap(),
"--db-url",
"sqlite::memory:",
"--disable-tls",
"--compute-driver",
"local",
"--compute-driver-socket",
"/tmp/local.sock",
]);
let result = super::prepare_server_config_with_drivers(
&mut args,
&matches,
&detected_local_registry(),
);
let Err(error) = result else {
panic!("network trust cannot be silently discarded by a remote endpoint");
};
assert!(error.to_string().contains("additional_ca_bundle"));
assert!(error.to_string().contains("unsupported remote endpoint"));
assert!(
!state.path().join("network-supervisor").exists(),
"rejected endpoint override must not stage a trust artifact"
);
}
#[test]
fn registry_detection_drives_auth_defaults_once() {
let state = tempfile::tempdir().unwrap();
crates/openshell-server/src/compute/driver_config.rs
+11 -1
View File
@@ -9,7 +9,7 @@
use crate::config_file;
use crate::defaults::LocalTlsPaths;
use openshell_core::{Error, Result};
use openshell_core::{Error, NetworkSupervisorTrustBundle, Result};
use serde::Deserialize;
use std::collections::BTreeMap;
use std::path::PathBuf;
@@ -67,6 +67,14 @@ impl GuestTlsPaths {
pub struct DriverStartupContext<'a> {
pub file: Option<&'a config_file::ConfigFile>,
pub guest_tls: Option<&'a GuestTlsPaths>,
/// A staged artifact for runtime driver construction. This is deliberately
/// absent during effective-config preflight.
pub network_trust: Option<&'a NetworkSupervisorTrustBundle>,
/// Whether destination trust sources were configured and normalized.
///
/// This remains available during preflight so selection can reject a
/// driver that cannot propagate trust, without fabricating an artifact.
pub network_trust_configured: bool,
pub gateway_port: u16,
pub gateway_tls_enabled: bool,
pub endpoint_overrides: &'a BTreeMap<String, PathBuf>,
@@ -216,6 +224,8 @@ mod tests {
DriverStartupContext {
file,
guest_tls: None,
network_trust: None,
network_trust_configured: false,
gateway_port: openshell_core::config::DEFAULT_SERVER_PORT,
gateway_tls_enabled: false,
endpoint_overrides,
crates/openshell-server/src/compute/mod.rs
+95 -18
View File
@@ -1216,6 +1216,11 @@ impl ComputeRuntime {
// Every start path derives resume from the durable transition, so a
// retried or recovered start repeats the request that was committed.
let resume = in_flight_resume_intent(sandbox);
// Starts may need to replace a stopped local workload/supervisor
// pair. Always reconstruct that input from the gateway's durable
// record; never let a driver derive provisioning input from mutable
// runtime inspection output.
let driver_sandbox = self.driver_sandbox_for_launch(sandbox)?;
Ok(StartSandboxRequest {
sandbox_id: sandbox.object_id().to_string(),
name: sandbox.object_name().to_string(),
@@ -1227,6 +1232,7 @@ impl ComputeRuntime {
isolation_backend: self.isolation_backend.clone(),
resume: resume.is_some(),
restart_if_restore_fails: resume.unwrap_or(false),
sandbox: Some(driver_sandbox),
})
}
@@ -2495,6 +2501,10 @@ impl ComputeRuntime {
} else {
None
};
// Starts may need to replace a stopped local workload/supervisor pair.
// The request (including its durable provisioning snapshot) is always
// reconstructed from the gateway's record; never let a driver derive
// provisioning input from mutable runtime inspection output.
let request =
self.driver_start_request(&starting, launch_authentication, expected_runtime_identity)?;
let driver = self.driver.clone();
@@ -4034,11 +4044,13 @@ impl ComputeRuntime {
/// Should be called once at gateway startup, before watchers spawn,
/// so the watch loop sees the post-start state on its first poll.
pub async fn start_persisted_sandboxes(&self) -> Result<(), String> {
self.start_persisted_sandboxes_with_authentication(
// The reconciliation futures carry full persisted sandbox records;
// keep them off the caller's stack (clippy::large_futures).
Box::pin(self.start_persisted_sandboxes_with_authentication(
|_| async { Ok(Vec::new()) },
|_| async { Ok(()) },
|_| {},
)
))
.await
}
@@ -4057,13 +4069,13 @@ impl ComputeRuntime {
authentication_failed: Failed,
) -> Result<(), String>
where
Authentication: Fn(&Sandbox) -> AuthenticationFuture,
AuthenticationFuture: Future<Output = Result<Vec<u8>, String>>,
Committed: Fn(&str) -> CommittedFuture,
CommittedFuture: Future<Output = Result<(), String>>,
Failed: Fn(&str),
Authentication: Fn(&Sandbox) -> AuthenticationFuture + Send + Sync,
AuthenticationFuture: Future<Output = Result<Vec<u8>, String>> + Send,
Committed: Fn(&str) -> CommittedFuture + Send + Sync,
CommittedFuture: Future<Output = Result<(), String>> + Send,
Failed: Fn(&str) + Send + Sync,
{
self.recover_persisted_lifecycle_transitions().await?;
Box::pin(self.recover_persisted_lifecycle_transitions()).await?;
if !self.driver_info.gateway_manages_lifecycle {
return Ok(());
}
@@ -4160,6 +4172,9 @@ impl ComputeRuntime {
continue;
}
};
// Recovery is allowed to reconstruct provisioning inputs only
// from the persisted gateway record (the request snapshot is
// rebuilt by `driver_start_request`).
let request = match self.driver_start_request(
&sandbox,
launch_authentication,
@@ -4420,8 +4435,8 @@ impl ComputeRuntime {
// Recovery retains the original attempt and deadline. A
// stalled driver must release this lifecycle gate after
// expiry so the deadline worker can reclaim its compute.
match self
.await_provisioning_operation(
match Box::pin(
self.await_provisioning_operation(
&sandbox,
self.driver.call_owned(
openshell_otel::rpc::START_SANDBOX,
@@ -4430,8 +4445,9 @@ impl ComputeRuntime {
driver.start_sandbox(Request::new(request)).await
},
),
)
.await
),
)
.await
{
Ok((_, settled)) => {
let updated = self.clear_resume_intent(*settled).await;
@@ -4492,6 +4508,11 @@ impl ComputeRuntime {
.map_err(|err| format!("failed to list sandboxes for {operation}: {err}"))
}
/// Fail closed when a start/recovery cannot reconstruct a driver's durable
/// provisioning snapshot. This is intentionally distinct from an
/// ambiguous transport failure: no start RPC has been issued, so keeping
/// `Starting` for a blind retry would allow a local reconciliation path to
/// fall back to mutable platform state.
async fn mark_sandbox_error(&self, sandbox: &Sandbox, reason: &str, message: &str) {
let _guard = self.lock_sandbox_local(sandbox.object_id()).await;
let sandbox_id = sandbox.object_id().to_string();
@@ -18073,11 +18094,31 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock"
));
driver.clear_calls();
runtime.start_persisted_sandboxes().await.unwrap();
runtime
.start_persisted_sandboxes_with_authentication(
|_| async { Ok(b"test-launch-authentication".to_vec()) },
|_| async { Ok(()) },
|_| {},
)
.await
.unwrap();
assert!(matches!(
driver.calls().as_slice(),
[FakeComputeDriverCall::GetCapabilities, FakeComputeDriverCall::StartSandbox { sandbox_id, sandbox_name, .. }]
if sandbox_id == "sb-uds" && sandbox_name == "uds-sandbox"
[
FakeComputeDriverCall::GetCapabilities,
FakeComputeDriverCall::StartSandbox {
sandbox_id,
sandbox_name,
launch_authentication,
sandbox: Some(durable),
..
},
]
if sandbox_id == "sb-uds"
&& sandbox_name == "uds-sandbox"
&& !launch_authentication.is_empty()
&& durable.id == "sb-uds"
&& durable.name == "uds-sandbox"
));
driver.clear_calls();
assert!(
@@ -18102,6 +18143,44 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock"
}
}
#[test]
fn start_sandbox_request_uses_optional_wire_field_nine_for_durable_snapshot() {
let legacy = StartSandboxRequest {
sandbox_id: "sandbox-id".into(),
name: "sandbox-name".into(),
launch_authentication: b"opaque-auth".to_vec(),
generation_id: "generation-id".into(),
expected_runtime_identity: String::new(),
isolation_backend: None,
resume: false,
restart_if_restore_fails: false,
sandbox: None,
};
let with_snapshot = StartSandboxRequest {
sandbox: Some(DriverSandbox {
id: "sandbox-id".into(),
name: "sandbox-name".into(),
..Default::default()
}),
..legacy.clone()
};
let legacy_wire = legacy.encode_to_vec();
let snapshot_wire = with_snapshot.encode_to_vec();
// Field 9 is length-delimited, hence its protobuf key is 0x4a. The
// legacy form does not emit it, preserving existing fields 1–8
// (fields 7 and 8 are the upstream suspend/resume controls).
assert!(!legacy_wire.contains(&0x4a));
assert!(snapshot_wire.contains(&0x4a));
assert_eq!(
StartSandboxRequest::decode(snapshot_wire.as_slice())
.unwrap()
.sandbox
.as_ref()
.map(|sandbox| sandbox.id.as_str()),
Some("sandbox-id")
);
}
#[tokio::test]
#[cfg(unix)]
async fn remote_compute_driver_rejects_policy_changes_before_forwarding() {
@@ -19992,9 +20071,7 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock"
runtime.store.put_message(&sandbox).await.unwrap();
let recovered_runtime = runtime.clone();
let mut recovery = tokio::spawn(async move {
recovered_runtime
.recover_persisted_lifecycle_transitions()
.await
Box::pin(recovered_runtime.recover_persisted_lifecycle_transitions()).await
});
tokio::time::timeout(Duration::from_secs(1), driver.start_started.notified())
.await
crates/openshell-server/src/config_file.rs
+88
View File
@@ -467,6 +467,22 @@ pub struct SupervisorFileSection {
/// operator-owned and changes require a gateway restart.
#[serde(default)]
pub middleware: Vec<MiddlewareServiceFileConfig>,
/// Shared network-supervisor configuration. This is intentionally outside
/// the compute-driver tables so every supported driver consumes the same
/// startup contract.
#[serde(default)]
pub network: SupervisorNetworkFileSection,
}
/// `[openshell.supervisor.network]` configuration.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct SupervisorNetworkFileSection {
/// Gateway-local pre-assembled PEM bundle whose certificates augment the
/// supervisor's destination trust roots. The gateway validates and stages
/// this single file during startup before any compute driver is constructed.
#[serde(default)]
pub additional_ca_bundle: Option<PathBuf>,
}
impl ConfigFile {
@@ -864,6 +880,19 @@ fn parse_and_validate(path: &Path, contents: &str) -> Result<ConfigFile, ConfigF
{
return Err(ConfigFileError::InvalidDriverTable { name: name.clone() });
}
if file
.openshell
.supervisor
.network
.additional_ca_bundle
.as_ref()
.is_some_and(|path| path.as_os_str().is_empty())
{
return Err(ConfigFileError::InvalidValue {
field: "openshell.supervisor.network.additional_ca_bundle",
message: "path must not be empty",
});
}
openshell_core::isolation_registration::validate_registrations(
&file.openshell.supervisor.isolation_backends,
@@ -1527,6 +1556,65 @@ allow_unauthenticated_users = true
assert!(auth.allow_unauthenticated_users);
}
#[test]
fn parses_supervisor_network_additional_ca_bundle() {
let tmp = write_tmp(
r#"
[openshell.supervisor.network]
additional_ca_bundle = "/etc/openshell/ca-bundle.crt"
"#,
);
let file = load(tmp.path()).expect("network supervisor config parses");
assert_eq!(
file.openshell.supervisor.network.additional_ca_bundle,
Some(PathBuf::from("/etc/openshell/ca-bundle.crt"))
);
}
#[test]
fn supervisor_network_additional_ca_bundle_defaults_to_unset() {
let file: ConfigFile = toml::from_str("[openshell.supervisor]\n").expect("valid config");
assert!(
file.openshell
.supervisor
.network
.additional_ca_bundle
.is_none()
);
}
#[test]
fn supervisor_network_rejects_unknown_fields() {
let tmp = write_tmp(
r"
[openshell.supervisor.network]
unknown = true
",
);
assert!(matches!(
load(tmp.path()),
Err(ConfigFileError::Parse { .. })
));
}
#[test]
fn supervisor_network_rejects_empty_ca_bundle_path() {
let tmp = write_tmp(
r#"
[openshell.supervisor.network]
additional_ca_bundle = ""
"#,
);
let error = load(tmp.path()).expect_err("empty source paths must fail");
assert!(
error
.to_string()
.contains("openshell.supervisor.network.additional_ca_bundle")
);
assert!(error.to_string().contains("must not be empty"));
}
#[test]
fn parses_supervisor_middleware_registration() {
let certificate =
crates/openshell-server/src/lib.rs
+181 -2
View File
@@ -31,6 +31,7 @@ mod grpc;
mod http;
mod middleware;
mod multiplex;
mod network_trust;
mod ocsf_log;
mod otel_tracing;
mod pagination;
@@ -62,7 +63,7 @@ mod ws_tunnel;
use openshell_core::net::set_tcp_nodelay_best_effort;
use openshell_core::telemetry::TelemetryComputeDriver;
use openshell_core::{Config, Error, ObjectLabels, Result};
use openshell_core::{Config, Error, NetworkSupervisorTrustBundle, ObjectLabels, Result};
use openshell_extension_core::{
BearerTokenSlot, ExtensionAudience, ExtensionCallerKind, ExtensionKind, MAX_EXTENSION_TOKEN_TTL,
};
@@ -301,6 +302,7 @@ pub(crate) struct ServerStartupConfig {
pub config: Config,
pub config_file: Option<config_file::ConfigFile>,
pub guest_tls: Option<compute::driver_config::GuestTlsPaths>,
pub network_trust: Option<NetworkSupervisorTrustBundle>,
pub compute_driver: ComputeDriverSelection,
pub legacy_compute_driver_env_seen: bool,
}
@@ -599,6 +601,7 @@ pub(crate) async fn run_server(
config,
config_file,
guest_tls,
network_trust,
compute_driver,
legacy_compute_driver_env_seen: _,
} = startup;
@@ -715,6 +718,8 @@ pub(crate) async fn run_server(
let driver_startup = compute::driver_config::DriverStartupContext {
file: config_file.as_ref(),
guest_tls: guest_tls.as_ref(),
network_trust: network_trust.as_ref(),
network_trust_configured: network_trust.is_some(),
gateway_port: config.bind_address.port(),
gateway_tls_enabled: config.tls.is_some(),
endpoint_overrides: &config.compute_driver_endpoints,
@@ -1441,6 +1446,16 @@ pub trait ComputeDriverFactory: Send + Sync {
Ok(Vec::new())
}
/// Whether this factory can propagate the gateway-owned destination trust
/// bundle to its network supervisor.
///
/// The default is deliberately conservative: out-of-tree, remote, and
/// otherwise unannotated drivers must explicitly opt in before a
/// configured additional CA bundle can be used with them.
fn supports_network_supervisor_trust(&self) -> bool {
false
}
async fn build(&self, context: ComputeDriverBuildContext<'_>) -> Result<ComputeDriverInstance>;
}
@@ -1788,6 +1803,13 @@ impl ComputeDriverBuildContext<'_> {
.map(compute::driver_config::GuestTlsPaths::as_path)
}
/// Gateway-normalized destination trust material for the network
/// supervisor. The bundle is absent when the global setting is omitted.
#[must_use]
pub fn network_trust_bundle(&self) -> Option<&NetworkSupervisorTrustBundle> {
self.config.driver_startup.network_trust
}
/// Deserialize the selected driver's merged TOML table.
pub fn driver_config<T>(&self) -> Result<T>
where
@@ -2046,6 +2068,14 @@ fn resolve_configured_compute_driver(
) -> Result<ConfiguredComputeDriver> {
let name = openshell_core::config::normalize_compute_driver_name(driver_name)
.map_err(Error::config)?;
let network_trust_configured =
driver_startup.network_trust.is_some() || driver_startup.network_trust_configured;
if network_trust_configured && driver_startup.endpoint_overrides.contains_key(&name) {
return Err(Error::config(format!(
"{} is configured but compute driver '{name}' uses an unsupported remote endpoint",
network_trust::CONFIG_FIELD
)));
}
// An operator-provided endpoint replaces normal construction for the
// selected name, including a compiled registration with the same name.
// The gateway connects to it; it does not provision the remote driver.
@@ -2054,9 +2084,21 @@ fn resolve_configured_compute_driver(
}
if let Some(registration) = registry.get(&name) {
if network_trust_configured && !registration.factory.supports_network_supervisor_trust() {
return Err(Error::config(format!(
"{} is configured but compute driver '{name}' does not support network trust propagation",
network_trust::CONFIG_FIELD
)));
}
return Ok(ConfiguredComputeDriver::Registered(registration.clone()));
}
if network_trust_configured {
return Err(Error::config(format!(
"{} is configured but remote compute driver '{name}' is unsupported because it cannot propagate network trust",
network_trust::CONFIG_FIELD
)));
}
Ok(ConfiguredComputeDriver::Remote { name })
}
@@ -2132,7 +2174,7 @@ mod tests {
serve_gateway_listener, validate_peer_endpoint_scheme,
};
use openshell_core::{
Config,
Config, NetworkSupervisorTrustBundle,
proto::{HealthRequest, open_shell_client::OpenShellClient},
};
use std::io::{Error, ErrorKind};
@@ -2319,16 +2361,35 @@ mod tests {
fn test_driver_startup<'a>(
config: &'a Config,
file: Option<&'a super::config_file::ConfigFile>,
) -> crate::compute::driver_config::DriverStartupContext<'a> {
test_driver_startup_with_bundle(config, file, None)
}
fn test_driver_startup_with_bundle<'a>(
config: &'a Config,
file: Option<&'a super::config_file::ConfigFile>,
network_trust: Option<&'a NetworkSupervisorTrustBundle>,
) -> crate::compute::driver_config::DriverStartupContext<'a> {
crate::compute::driver_config::DriverStartupContext {
file,
guest_tls: None,
network_trust,
network_trust_configured: network_trust.is_some(),
gateway_port: openshell_core::config::DEFAULT_SERVER_PORT,
gateway_tls_enabled: false,
endpoint_overrides: &config.compute_driver_endpoints,
}
}
fn test_network_trust_bundle() -> NetworkSupervisorTrustBundle {
NetworkSupervisorTrustBundle::new(
b"-----BEGIN CERTIFICATE-----\nfixture\n-----END CERTIFICATE-----\n".to_vec(),
1,
"sha256:fixture",
"/tmp/openshell-additional-ca.crt".into(),
)
}
fn test_compute_drivers() -> super::ComputeDriverRegistry {
let mut registry = super::ComputeDriverRegistry::new();
for (name, priority) in [("alpha", 100), ("beta", 200), ("gamma", 300)] {
@@ -2378,6 +2439,8 @@ mod tests {
guest_tls: None,
gateway_port: 8080,
gateway_tls_enabled: true,
network_trust: None,
network_trust_configured: false,
endpoint_overrides: &endpoint_overrides,
},
},
@@ -2415,6 +2478,8 @@ mod tests {
driver_startup: crate::compute::driver_config::DriverStartupContext {
file: None,
guest_tls: None,
network_trust: None,
network_trust_configured: false,
gateway_port: 8080,
gateway_tls_enabled: false,
endpoint_overrides: &endpoint_overrides,
@@ -2981,6 +3046,120 @@ mod tests {
));
}
#[test]
fn network_trust_is_available_to_startup_context() {
let config = Config::new(None).with_compute_driver("docker");
let bundle = test_network_trust_bundle();
let startup = test_driver_startup_with_bundle(&config, None, Some(&bundle));
assert_eq!(
startup
.network_trust
.expect("configured trust bundle")
.certificate_count(),
1
);
}
#[test]
fn configured_network_trust_rejects_endpoint_override_before_remote_connection() {
let config = Config::new(None)
.with_compute_driver("docker")
.with_compute_driver_endpoint("docker", "/run/openshell/docker.sock");
let bundle = test_network_trust_bundle();
let error = configured_compute_driver(
&test_compute_drivers(),
&config,
test_driver_startup_with_bundle(&config, None, Some(&bundle)),
)
.expect_err("destination trust must not be discarded by a remote override");
assert!(error.to_string().contains("additional_ca_bundle"));
assert!(error.to_string().contains("unsupported"));
assert!(!error.to_string().contains("docker.sock"));
}
#[test]
fn configured_network_trust_rejects_unknown_remote_driver() {
let config = Config::new(None).with_compute_driver("kyma");
let bundle = test_network_trust_bundle();
let error = configured_compute_driver(
&test_compute_drivers(),
&config,
test_driver_startup_with_bundle(&config, None, Some(&bundle)),
)
.expect_err("destination trust must not be silently ignored by a remote driver");
assert!(error.to_string().contains("additional_ca_bundle"));
assert!(error.to_string().contains("kyma"));
}
#[test]
fn configured_network_trust_rejects_unsupported_custom_registration() {
let mut registry = super::ComputeDriverRegistry::new();
registry
.install(
super::ComputeDriverRegistration::new("custom", 1, None, TestComputeDriverFactory)
.expect("custom registration"),
)
.expect("install custom registration");
let config = Config::new(None).with_compute_driver("custom");
let bundle = test_network_trust_bundle();
let error = configured_compute_driver(
&registry,
&config,
test_driver_startup_with_bundle(&config, None, Some(&bundle)),
)
.expect_err("custom drivers cannot consume the shared trust contract yet");
assert!(error.to_string().contains("additional_ca_bundle"));
assert!(error.to_string().contains("custom"));
assert!(error.to_string().contains("network trust propagation"));
}
#[derive(Clone, Copy)]
struct TrustPropagatingTestFactory;
#[async_trait::async_trait]
impl super::ComputeDriverFactory for TrustPropagatingTestFactory {
fn supports_network_supervisor_trust(&self) -> bool {
true
}
async fn build(
&self,
_context: super::ComputeDriverBuildContext<'_>,
) -> openshell_core::Result<super::ComputeDriverInstance> {
unreachable!("selection tests do not construct the driver")
}
}
#[test]
fn configured_network_trust_accepts_explicitly_capable_registration() {
let mut registry = super::ComputeDriverRegistry::new();
registry
.install(
super::ComputeDriverRegistration::new(
"custom",
1,
None,
TrustPropagatingTestFactory,
)
.expect("custom registration"),
)
.expect("install custom registration");
let config = Config::new(None).with_compute_driver("custom");
let bundle = test_network_trust_bundle();
let driver = configured_compute_driver(
&registry,
&config,
test_driver_startup_with_bundle(&config, None, Some(&bundle)),
)
.expect("an explicitly capable registration accepts destination trust");
assert!(matches!(driver, ConfiguredComputeDriver::Registered(_)));
}
#[tokio::test]
async fn failed_gateway_listener_bind_does_not_attempt_persisted_sandbox_start() {
let occupied_listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
crates/openshell-server/src/network_trust.rs
+1147
View File
File diff suppressed because it is too large Load Diff
crates/openshell-server/src/test_support.rs
+9
View File
@@ -113,7 +113,14 @@ pub enum FakeComputeDriverCall {
sandbox_id: String,
sandbox_name: String,
isolation_backend: Option<openshell_core::proto::compute::v1::IsolationBackendRegistration>,
/// Fresh supervisor-facing credentials. These remain opaque in test
/// diagnostics just as they do in production tracing.
launch_authentication: Vec<u8>,
generation_id: String,
/// The optional durable provisioning snapshot supplied by newer
/// gateways. Keeping it in the fake makes compatibility and
/// reconciliation tests able to verify the start contract.
sandbox: Option<DriverSandbox>,
},
DeleteSandbox {
sandbox_id: String,
@@ -437,6 +444,8 @@ impl ComputeDriver for FakeComputeDriver {
sandbox_name: request.name,
isolation_backend: request.isolation_backend,
launch_authentication: request.launch_authentication,
generation_id: request.generation_id,
sandbox: request.sandbox,
});
});
Ok(Response::new(StartSandboxResponse::default()))
deploy/helm/openshell/README.md
+72
View File
@@ -211,6 +211,77 @@ See [`values.yaml`](values.yaml) for source defaults. Selected overlays:
- [`ci/values-spire.yaml`](ci/values-spire.yaml) - SPIFFE/SPIRE provider token grants
- [`ci/values-spire-stack.yaml`](ci/values-spire-stack.yaml) - SPIRE hardened chart values for local development
### Additional destination CAs
Create an operator-owned ConfigMap with a `ca.crt` key when sandboxes must reach
TLS destinations signed by a private CA:
```shell
kubectl create configmap private-destination-ca -n openshell \
--from-file=ca.crt=/path/to/ca.crt
```
Then set the source ConfigMap name:
```yaml
supervisor:
network:
additionalCaConfigMapName: private-destination-ca
```
The chart mounts the source into the gateway and renders
`[openshell.supervisor.network].additional_ca_bundle`. The gateway validates
and normalizes the one certificate-only PEM bundle at startup. The Kubernetes
driver creates a separate managed ConfigMap in each target sandbox namespace
and mounts it only in the dedicated supervisor Pod at
`/etc/openshell/tls/additional/ca-bundle.pem`. That supervisor receives the
path through `--additional-ca-bundle` and folds the bundle into system roots.
The configured roots augment default destination trust. They do not configure
the corporate proxy, OIDC, or gateway callback mTLS trust.
Changing only the source ConfigMap does not roll or reload the gateway. Restart
the gateway, then recreate or restart affected sandboxes to use the new roots.
Removing `additionalCaConfigMapName` removes the global TOML setting; newly
created or recreated sandbox pods then have no additional-CA volume or mount.
Existing running sandbox supervisors retain their startup trust until they are
restarted or recreated.
The driver creates one immutable, content-addressed ConfigMap generation in
each target namespace. Names begin with
`openshell-additional-ca-` and include DNS-safe hashes of the effective
gateway ID and normalized bundle generation. Existing objects are accepted only
when their ownership labels, generation annotation, immutability, and exact
`ca.crt` data match; the driver never patches or updates them.
Kubernetes cannot restrict `create` by `resourceNames`, and `resourceNames`
cannot express the digest-bearing name prefix needed for `get`. Enabling this
feature therefore gives the gateway/driver service account namespace-wide
ConfigMap `get` and `create` permissions in shared mode. In managed/operator
modes, the chart's ClusterRoleBinding makes those permissions cluster-wide for
the bound service account. The chart does not grant ConfigMap `list`, `watch`,
`patch`, `update`, or `delete`. Use a dedicated gateway service account and
namespace-scoped bindings when this boundary matters. If you render the TOML
setting outside this Helm value, grant `get` and `create` in every target
sandbox namespace.
Disabling the setting intentionally does not delete driver-managed generations:
the gateway has no ConfigMap list/delete permissions, and an older object may
still be referenced by a running or stopped sandbox. An operator with separate
list/delete authority can inventory the gateway's generations by label:
```shell
kubectl -n <sandbox-namespace> get configmap \
-l 'openshell.ai/managed-by=openshell,openshell.ai/gateway-id=<effective-gateway-id>'
```
Before deleting a named generation, verify that no Sandbox resource or pod
volume in that namespace references it. Delete only the confirmed unreferenced
name; removing a referenced generation can make a later pod start or reschedule
fail closed.
A configured CA can authenticate every policy-permitted TLS endpoint with a
matching certificate chain and hostname. Scope private CAs accordingly.
### Database backend
By default, OpenShell uses SQLite and runs the gateway as a StatefulSet so the
@@ -626,6 +697,7 @@ discovery endpoint or its TLS CA.
| supervisor.image.registry | string | `""` | Supervisor image registry. Empty uses global.image.registry. |
| supervisor.image.repository | string | `"openshell/supervisor"` | Supervisor image repository. |
| supervisor.image.tag | string | `""` | Supervisor image tag. Defaults to the chart appVersion when empty. |
| supervisor.network.additionalCaConfigMapName | string | `""` | Name of an operator-owned ConfigMap with a `ca.crt` key containing additional destination CA certificates for sandbox supervisors. Empty disables additional destination trust. |
| supervisor.podDisruptionBudget.enabled | bool | `true` | Render a PodDisruptionBudget with maxUnavailable 1 for the shared supervisor, used only when gatewayConfig selects supervisor_topology = "gateway". Disable it only when maintenance must evict more than one supervisor Pod at a time, or evict a member while another is unavailable. |
| supervisor.podSecurityContext.fsGroup | int | `65534` | fsGroup assigned to the shared supervisor pod. |
| supervisor.podSecurityContext.runAsGroup | int | `65534` | Primary GID assigned to the shared supervisor pod. |
deploy/helm/openshell/README.md.gotmpl
+71
View File
@@ -212,6 +212,77 @@ See [`values.yaml`](values.yaml) for source defaults. Selected overlays:
- [`ci/values-spire.yaml`](ci/values-spire.yaml) - SPIFFE/SPIRE provider token grants
- [`ci/values-spire-stack.yaml`](ci/values-spire-stack.yaml) - SPIRE hardened chart values for local development
### Additional destination CAs
Create an operator-owned ConfigMap with a `ca.crt` key when sandboxes must reach
TLS destinations signed by a private CA:
```shell
kubectl create configmap private-destination-ca -n openshell \
--from-file=ca.crt=/path/to/ca.crt
```
Then set the source ConfigMap name:
```yaml
supervisor:
network:
additionalCaConfigMapName: private-destination-ca
```
The chart mounts the source into the gateway and renders
`[openshell.supervisor.network].additional_ca_bundle`. The gateway validates
and normalizes the one certificate-only PEM bundle at startup. The Kubernetes
driver creates a separate managed ConfigMap in each target sandbox namespace
and mounts it only in the dedicated supervisor Pod at
`/etc/openshell/tls/additional/ca-bundle.pem`. That supervisor receives the
path through `--additional-ca-bundle` and folds the bundle into system roots.
The configured roots augment default destination trust. They do not configure
the corporate proxy, OIDC, or gateway callback mTLS trust.
Changing only the source ConfigMap does not roll or reload the gateway. Restart
the gateway, then recreate or restart affected sandboxes to use the new roots.
Removing `additionalCaConfigMapName` removes the global TOML setting; newly
created or recreated sandbox pods then have no additional-CA volume or mount.
Existing running sandbox supervisors retain their startup trust until they are
restarted or recreated.
The driver creates one immutable, content-addressed ConfigMap generation in
each target namespace. Names begin with
`openshell-additional-ca-` and include DNS-safe hashes of the effective
gateway ID and normalized bundle generation. Existing objects are accepted only
when their ownership labels, generation annotation, immutability, and exact
`ca.crt` data match; the driver never patches or updates them.
Kubernetes cannot restrict `create` by `resourceNames`, and `resourceNames`
cannot express the digest-bearing name prefix needed for `get`. Enabling this
feature therefore gives the gateway/driver service account namespace-wide
ConfigMap `get` and `create` permissions in shared mode. In managed/operator
modes, the chart's ClusterRoleBinding makes those permissions cluster-wide for
the bound service account. The chart does not grant ConfigMap `list`, `watch`,
`patch`, `update`, or `delete`. Use a dedicated gateway service account and
namespace-scoped bindings when this boundary matters. If you render the TOML
setting outside this Helm value, grant `get` and `create` in every target
sandbox namespace.
Disabling the setting intentionally does not delete driver-managed generations:
the gateway has no ConfigMap list/delete permissions, and an older object may
still be referenced by a running or stopped sandbox. An operator with separate
list/delete authority can inventory the gateway's generations by label:
```shell
kubectl -n <sandbox-namespace> get configmap \
-l 'openshell.ai/managed-by=openshell,openshell.ai/gateway-id=<effective-gateway-id>'
```
Before deleting a named generation, verify that no Sandbox resource or pod
volume in that namespace references it. Delete only the confirmed unreferenced
name; removing a referenced generation can make a later pod start or reschedule
fail closed.
A configured CA can authenticate every policy-permitted TLS endpoint with a
matching certificate chain and hostname. Scope private CAs accordingly.
### Database backend
By default, OpenShell uses SQLite and runs the gateway as a StatefulSet so the
deploy/helm/openshell/templates/_gateway-workload.tpl
+26 -1
View File
@@ -17,12 +17,14 @@ Gateway pod template shared by the StatefulSet and Deployment workload shapes.
{{- $legacyCredentialDrivers := .Values.server.credentialDrivers | default dict -}}
{{- $legacyVaultResources := get $legacyCredentialDrivers "vault" | default dict -}}
{{- $vaultCaConfigMapName := get $vaultResources "caConfigMapName" | default (get $legacyVaultResources "caConfigMapName") -}}
{{- $supervisorNetwork := .Values.supervisor.network | default dict -}}
metadata:
annotations:
# Roll the gateway workload when the rendered gateway TOML changes - the
# gateway only reads /etc/openshell/gateway.toml at startup, so without
# this annotation a `helm upgrade` that only mutates the ConfigMap would
# leave pods running with stale config.
# leave pods running with stale config. The operator-owned additional-CA
# ConfigMap is not checksummed; restart the gateway after changing its data.
checksum/gateway-config: {{ include (print $.Template.BasePath "/gateway-config.yaml") . | sha256sum }}
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 4 }}
@@ -134,6 +136,11 @@ spec:
- name: SSL_CERT_FILE
value: /etc/openshell-tls/oidc-ca/ca.crt
{{- end }}
# The gateway stages normalized, content-addressed supervisor trust
# artifacts under its XDG state directory. Keep that state on a
# dedicated writable volume for both StatefulSet and Deployment modes.
- name: XDG_STATE_HOME
value: /var/lib/openshell-state
- name: OPENSHELL_TELEMETRY_ENABLED
value: {{ .Values.server.telemetryEnabled | quote }}
{{- if $spiffeSocketPath }}
@@ -161,6 +168,12 @@ spec:
- name: gateway-peer-token
mountPath: /var/run/secrets/openshell-peer
readOnly: true
{{- if get $supervisorNetwork "additionalCaConfigMapName" }}
- name: additional-ca-source
mountPath: /etc/openshell-tls/additional-ca-source/ca.crt
subPath: ca.crt
readOnly: true
{{- end }}
{{- if not .Values.server.disableTls }}
- name: tls-cert
mountPath: /etc/openshell-tls/server
@@ -202,6 +215,8 @@ spec:
{{- with .Values.server.extraVolumeMounts }}
{{- toYaml . | nindent 8 }}
{{- end }}
- name: gateway-state
mountPath: /var/lib/openshell-state
ports:
- name: grpc
containerPort: {{ .Values.service.port }}
@@ -255,6 +270,14 @@ spec:
path: token
audience: openshell-gateway-peer
expirationSeconds: 3600
{{- if get $supervisorNetwork "additionalCaConfigMapName" }}
- name: additional-ca-source
configMap:
name: {{ get $supervisorNetwork "additionalCaConfigMapName" | quote }}
items:
- key: ca.crt
path: ca.crt
{{- end }}
{{- if not .Values.server.disableTls }}
- name: tls-cert
secret:
@@ -310,6 +333,8 @@ spec:
{{- with .Values.server.extraVolumes }}
{{- toYaml . | nindent 4 }}
{{- end }}
- name: gateway-state
emptyDir: {}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 4 }}
deploy/helm/openshell/templates/_toml.tpl
+10
View File
@@ -315,6 +315,16 @@ openshell.supervisorPoolConfig. */}}
{{- if $supervisorPool -}}
{{- $_ := set $config "openshell.gateway.supervisor_pool" $supervisorPool -}}
{{- end -}}
{{/* Additional destination CA trust is chart plumbing: the runtime table is
owned by the chart, not gatewayConfig. Derive it only from the chart-owned
operator ConfigMap reference. */}}
{{- $supervisorNetwork := .Values.supervisor.network | default dict -}}
{{- if get $supervisorNetwork "additionalCaConfigMapName" -}}
{{- if hasKey $config "openshell.supervisor.network" -}}
{{- fail "gatewayConfig must not define openshell.supervisor.network; the chart derives it from supervisor.network.additionalCaConfigMapName" -}}
{{- end -}}
{{- $_ := set $config "openshell.supervisor.network" (dict "additional_ca_bundle" "/etc/openshell-tls/additional-ca-source/ca.crt") -}}
{{- end -}}
{{/* A nested YAML map and its dotted top-level table would serialize to
conflicting TOML definitions. Fail before rendering an invalid ConfigMap. */}}
{{- range $tableName := keys $config | sortAlpha -}}
deploy/helm/openshell/templates/clusterrole.yaml
+18
View File
@@ -4,6 +4,7 @@
{{ $kubernetesConfig := include "openshell.effectiveKubernetesConfig" . | fromYaml -}}
{{- $workspaceMode := get $kubernetesConfig "workspace_mode" | default "shared" }}
{{- $supervisorNetwork := .Values.supervisor.network | default dict }}
{{- $allowDriverConfig := get $kubernetesConfig "allow_driver_config" | default false -}}
{{- $managedSshIngressEnabled := .Values.networkPolicy.enabled -}}
{{- $gatewayConfig := .Values.gatewayConfig | default dict -}}
@@ -98,6 +99,23 @@ rules:
- patch
- watch
{{- end }}
{{- if and (ne $workspaceMode "shared") (get $supervisorNetwork "additionalCaConfigMapName") }}
# The Kubernetes driver creates and verifies immutable, content-addressed
# destination-CA ConfigMaps. Kubernetes RBAC supports exact resourceNames
# only, not the digest-bearing name prefix, and CREATE cannot be restricted
# by resourceNames. This ClusterRole is bound cluster-wide in managed and
# operator modes, so keep the required capability to get/create only: the
# driver never lists, watches, patches, updates, or deletes these objects.
# Use a dedicated gateway service account and namespace-scoped bindings when
# this read boundary matters.
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- create
{{- end }}
{{- if ne $workspaceMode "shared" }}
- apiGroups: [""]
resources: ["services"]
deploy/helm/openshell/templates/role.yaml
+13 -1
View File
@@ -1,7 +1,9 @@
{{- $kubernetesConfig := include "openshell.effectiveKubernetesConfig" . | fromYaml -}}
{{- $workspaceMode := get $kubernetesConfig "workspace_mode" | default "shared" -}}
{{- $allowDriverConfig := get $kubernetesConfig "allow_driver_config" | default false -}}
{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) (include "openshell.rbacCreate" .) }}
{{- $supervisorNetwork := .Values.supervisor.network | default dict -}}
{{- $additionalCaEnabled := ne (get $supervisorNetwork "additionalCaConfigMapName" | default "") "" -}}
{{- if and (eq $workspaceMode "shared") (or (include "openshell.workspaceResourcesEnabled" .) $additionalCaEnabled) (include "openshell.rbacCreate" .) }}
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
apiVersion: rbac.authorization.k8s.io/v1
@@ -12,6 +14,7 @@ metadata:
labels:
{{- include "openshell.labels" . | nindent 4 }}
rules:
{{- if include "openshell.workspaceResourcesEnabled" . }}
{{- if $allowDriverConfig }}
# Metadata-only admission of caller-selected PVCs requires get permission.
- apiGroups: [""]
@@ -67,4 +70,13 @@ rules:
- apiGroups: ["networking.k8s.io"]
resources: ["networkpolicies"]
verbs: ["create", "get"]
{{- end }}
{{- if $additionalCaEnabled }}
# Immutable, content-addressed destination-CA generations are verified by
# exact name. The driver intentionally has no list/watch/patch/update/delete
# capability for ConfigMaps, so retained generations cannot be mutated.
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "create"]
{{- end }}
{{- end }}
deploy/helm/openshell/templates/rolebinding.yaml
+3 -1
View File
@@ -1,6 +1,8 @@
{{- $kubernetesConfig := include "openshell.effectiveKubernetesConfig" . | fromYaml -}}
{{- $workspaceMode := get $kubernetesConfig "workspace_mode" | default "shared" -}}
{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) (include "openshell.rbacCreate" .) }}
{{- $supervisorNetwork := .Values.supervisor.network | default dict -}}
{{- $additionalCaEnabled := ne (get $supervisorNetwork "additionalCaConfigMapName" | default "") "" -}}
{{- if and (eq $workspaceMode "shared") (include "openshell.rbacCreate" .) (or (include "openshell.workspaceResourcesEnabled" .) $additionalCaEnabled) }}
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
apiVersion: rbac.authorization.k8s.io/v1
deploy/helm/openshell/tests/gateway_config_test.yaml
+29
View File
@@ -1294,6 +1294,25 @@ tests:
- failedTemplate:
errorPattern: "replicaCount > 1 requires server.externalDbSecret"
- it: renders writable ephemeral XDG state for gateway trust artifacts
template: templates/statefulset.yaml
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: XDG_STATE_HOME
value: /var/lib/openshell-state
- contains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: gateway-state
mountPath: /var/lib/openshell-state
- contains:
path: spec.template.spec.volumes
content:
name: gateway-state
emptyDir: {}
- it: renders a Deployment for external database-backed gateway workloads
template: templates/deployment.yaml
set:
@@ -1331,6 +1350,16 @@ tests:
secretKeyRef:
name: my-pg-secret
key: uri
- contains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: gateway-state
mountPath: /var/lib/openshell-state
- contains:
path: spec.template.spec.volumes
content:
name: gateway-state
emptyDir: {}
- it: fails when a Deployment uses the default SQLite database
template: templates/statefulset.yaml
deploy/helm/openshell/tests/network_additional_ca_test.yaml
+223
View File
@@ -0,0 +1,223 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
suite: network additional CA delivery
release:
name: openshell
namespace: my-namespace
templates:
- templates/gateway-config.yaml
- templates/statefulset.yaml
- templates/role.yaml
- templates/rolebinding.yaml
- templates/clusterrole.yaml
tests:
- it: omits the global additional CA setting by default
template: templates/gateway-config.yaml
asserts:
- notMatchRegex:
path: data["gateway.toml"]
pattern: '\[openshell\.supervisor\.network\]'
- it: treats a missing supervisor network map as disabled
template: templates/gateway-config.yaml
set:
supervisor.network: null
asserts:
- notMatchRegex:
path: data["gateway.toml"]
pattern: '\[openshell\.supervisor\.network\]'
- it: omits the gateway source mount by default
template: templates/statefulset.yaml
asserts:
- notContains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: additional-ca-source
any: true
- notContains:
path: spec.template.spec.volumes
content:
name: additional-ca-source
any: true
- it: omits shared namespace ConfigMap permissions by default
template: templates/role.yaml
asserts:
- notContains:
path: rules
content:
apiGroups: [""]
resources: ["configmaps"]
any: true
- it: omits cluster-wide ConfigMap permissions by default
template: templates/clusterrole.yaml
asserts:
- notContains:
path: rules
content:
apiGroups: [""]
resources: ["configmaps"]
any: true
- it: renders the global setting outside the Kubernetes driver table
template: templates/gateway-config.yaml
set:
supervisor.network.additionalCaConfigMapName: private-destination-ca
asserts:
- matchRegex:
path: data["gateway.toml"]
pattern: '(?ms)\[openshell\.supervisor\.network\]\s*additional_ca_bundle\s*=\s*"/etc/openshell-tls/additional-ca-source/ca\.crt"'
- notMatchRegex:
path: data["gateway.toml"]
pattern: '(?ms)\[openshell\.drivers\.kubernetes\][^\[]*additional_ca_bundle'
- it: mounts the operator source ConfigMap into the gateway
template: templates/statefulset.yaml
set:
supervisor.network.additionalCaConfigMapName: private-destination-ca
asserts:
- contains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: additional-ca-source
mountPath: /etc/openshell-tls/additional-ca-source/ca.crt
subPath: ca.crt
readOnly: true
- contains:
path: spec.template.spec.volumes
content:
name: additional-ca-source
configMap:
name: private-destination-ca
items:
- key: ca.crt
path: ca.crt
- exists:
path: spec.template.metadata.annotations["checksum/gateway-config"]
- it: grants only namespace-wide ConfigMap get and create in shared mode
template: templates/role.yaml
set:
supervisor.network.additionalCaConfigMapName: private-destination-ca
workspaceResources.enabled: false
asserts:
- contains:
path: rules
content:
apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "create"]
- notContains:
path: rules
content:
apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["openshell-additional-ca-openshell"]
any: true
- notContains:
path: rules
content:
apiGroups: ["agents.x-k8s.io"]
any: true
- it: binds shared namespace ConfigMap permissions when workspace resources are external
template: templates/rolebinding.yaml
set:
supervisor.network.additionalCaConfigMapName: private-destination-ca
workspaceResources.enabled: false
asserts:
- hasDocuments:
count: 1
- it: grants only ConfigMap get/create when shared workspace resources are external
template: templates/role.yaml
set:
supervisor.network.additionalCaConfigMapName: private-destination-ca
workspaceResources.enabled: false
asserts:
- contains:
path: rules
content:
apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "create"]
- notContains:
path: rules
content:
apiGroups: [""]
resources: ["pods"]
any: true
- notContains:
path: rules
content:
apiGroups: [""]
resources: ["secrets"]
any: true
- notContains:
path: rules
content:
apiGroups: ["networking.k8s.io"]
resources: ["networkpolicies"]
any: true
- it: grants only cluster-wide ConfigMap get and create in managed mode
template: templates/clusterrole.yaml
set:
supervisor.network.additionalCaConfigMapName: private-destination-ca
server.drivers.kubernetes.workspaceMode: managed
asserts:
- contains:
path: rules
content:
apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "create"]
- notContains:
path: rules
content:
apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["openshell-additional-ca-openshell"]
any: true
- it: omits a shared Role in managed workspace mode
template: templates/role.yaml
set:
supervisor.network.additionalCaConfigMapName: private-destination-ca
server.drivers.kubernetes.workspaceMode: managed
asserts:
- hasDocuments:
count: 0
- it: grants only cluster-wide ConfigMap get and create in operator mode
template: templates/clusterrole.yaml
set:
supervisor.network.additionalCaConfigMapName: private-destination-ca
server.drivers.kubernetes.workspaceMode: operator
asserts:
- contains:
path: rules
content:
apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "create"]
- notContains:
path: rules
content:
apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["openshell-additional-ca-openshell"]
any: true
- it: omits a shared Role in operator workspace mode
template: templates/role.yaml
set:
supervisor.network.additionalCaConfigMapName: private-destination-ca
server.drivers.kubernetes.workspaceMode: operator
asserts:
- hasDocuments:
count: 0
deploy/helm/openshell/values.yaml
+3
View File
@@ -134,6 +134,9 @@ supervisor:
# gatewayConfig selects supervisor_topology = "gateway". Top-level affinity
# applies only to the gateway pod.
affinity: {}
network:
# -- Name of an operator-owned ConfigMap with a `ca.crt` key containing additional destination CA certificates for sandbox supervisors. Empty disables additional destination trust.
additionalCaConfigMapName: ""
sandboxRuntime:
# -- Workload boundary TLS listener port.
boundaryPort: 5500
docs/how-it-works/gateways/configuration.mdx
+97 -16
View File
@@ -90,6 +90,9 @@ version = 2
[openshell.gateway.oidc]
# ... JWT bearer auth ...
[openshell.supervisor.network]
# ... shared network-supervisor destination trust ...
[openshell.drivers.kubernetes]
# ... driver-specific settings ...
@@ -99,6 +102,95 @@ version = 2
The gateway selector is `compute_driver = "<driver>"`. It accepts one scalar driver name. Omit it to retain auto-detection.
## Additional Destination CAs
Use the global supervisor network table to trust private certificate authorities
for sandbox egress destinations:
```toml
[openshell.supervisor.network]
additional_ca_bundle = "/etc/openshell/certs/private-destination-ca.pem"
```
The path refers to one pre-assembled PEM bundle on the gateway host or in the
gateway container. It must contain one or more PEM certificate blocks. Inline
PEM, private keys, DER-only input, URLs, empty files, unrelated text, mixed
certificate/key files, symlinks, and non-regular files are unsupported. The
gateway reads the bundle once at startup, validates and canonicalizes it into a
gateway-owned artifact, and fails startup with the field and source path when
material is missing or unusable. It does not include certificate contents in
diagnostics.
The normalized certificates augment the default bundled, system, or native
roots used for sandbox destination TLS. Normal certificate-chain and hostname
verification remains enabled. A configured CA can authenticate every
policy-permitted endpoint whose certificate chains to that CA and matches the
requested hostname, so scope private CAs accordingly.
This setting is separate from all other trust domains:
- `proxy_ca_bundle` authenticates a configured corporate proxy and its TLS
interception path. It still requires the corresponding proxy setting.
- `guest_tls_*`, `OPENSHELL_TLS_CA`, and Kubernetes client TLS Secrets
authenticate sandbox callbacks to the gateway.
- `[openshell.gateway.tls]` configures the gateway listener.
- OIDC, middleware, and interceptor CA settings authenticate their own
gateway-side connections.
Docker and Podman bind-mount the gateway-owned artifact, Kubernetes creates an
immutable content-addressed managed ConfigMap, and VM stages its snapshot in the
sandbox overlay. Each mounts it at
`/etc/openshell/tls/additional/ca-bundle.pem` and starts the supervisor with
`--additional-ca-bundle` for that path. Drivers verify their staged artifact
before handing it to a runtime; the SHA-256 generation is an internal staging
value, not a supervisor argument. The supervisor
reads the mounted bundle and folds it into its system roots. Remote and custom
compute drivers reject this setting because the compute-driver protocol does
not carry the normalized material.
### Lifecycle and certificate rollover
This is **global gateway configuration**, not a per-sandbox setting. Source
files (or the Helm source ConfigMap) are read and normalized **only when the
gateway starts**. Updating a source file or ConfigMap alone has no effect:
OpenShell does not watch or hot-reload those sources, and it does not
automatically restart sandbox consumers. A running supervisor retains the
startup trust bundle it already loaded.
To apply any addition, removal, or source update, the operator must:
1. Update the source files or source ConfigMap.
2. Restart or redeploy the gateway so it reads and normalizes the new source.
3. Explicitly stop and then start **every affected sandbox**.
Stop/start retains the sandbox identity and its supported durable workspace; it
does not create a replacement sandbox. The restarted supervisor consumes the
bundle staged from the restarted gateway's snapshot. Source normalization,
driver delivery, and supervisor validation failures fail closed:
they do not silently fall back to a previous or default trust bundle.
Use this sequence for a non-disruptive private-CA rollover:
1. Configure both the old and new roots.
2. Restart or redeploy the gateway, then stop/start every affected sandbox.
3. Rotate destination certificates to chain to the new root.
4. Remove the old root, then repeat the gateway restart/redeploy and
stop/start sequence for every affected sandbox.
For Helm, create an operator-owned ConfigMap with a `ca.crt` key and set
`supervisor.network.additionalCaConfigMapName`. The chart mounts the source at
`/etc/openshell-tls/additional-ca-source/ca.crt`, renders that path in
the global table, and grants the Kubernetes driver ConfigMap `get` and `create`
permissions for the selected workspace mode. These permissions cover the shared
sandbox namespace in shared mode and are cluster-wide through the existing
ClusterRoleBinding in managed and operator modes. Kubernetes RBAC cannot limit
`create` by resource name or limit `get` to the digest-bearing name prefix. The
chart does not grant ConfigMap `list`, `watch`, `patch`, `update`, or `delete`
for this feature. Use a dedicated gateway service account and namespace-scoped
bindings when this read boundary matters. Updating only the source ConfigMap
does not trigger the gateway workload checksum, so explicitly restart or
redeploy the gateway and then stop/start every affected sandbox.
## Migrate to schema version 2
Schema version 2 is an intentional breaking cutover. The gateway rejects files
@@ -229,6 +321,11 @@ provider_profile_sources = [
{ type = "user" },
]
# Additional private destination roots are global supervisor settings, not
# compute-driver settings. The gateway reads certificate-only PEM at startup.
[openshell.supervisor.network]
additional_ca_bundle = "/etc/openshell/certs/private-destination-ca.pem"
# Operator-run supervisor middleware. The gRPC endpoint must be reachable from
# both the gateway and sandbox supervisors.
[[openshell.supervisor.middleware]]
@@ -1379,22 +1476,6 @@ health_check_interval_secs = 10
# proxy_connect_by_hostname = true
# Corporate CA trusted for an https:// proxy and TLS-intercepting proxies.
# proxy_ca_bundle = "/etc/openshell/tls/proxy-ca.pem"
# additional_ca_bundle points at a PEM CA bundle (on the gateway host) trusted
# for direct, policy-inspected HTTPS egress to a destination signed by a
# private CA — independent of proxy_ca_bundle above, this applies whether or
# not https_proxy is set. The Podman supervisor runs as its own container,
# separate from the workload image, so it cannot inherit a private CA baked
# into the workload's own trust store; this setting restores that trust
# without rebuilding the supervisor image. The gateway bind-mounts the file
# read-only into the supervisor container and folds it into the upstream
# trust store alongside (not instead of) the system CA bundle. The CA is also
# added to the trust bundle inside each sandbox, so workloads trust it directly.
# It is trusted for every destination, not only the one that needs it, so use a
# CA you control or one with name constraints. The file must
# exist and contain at least one certificate; the gateway validates this on
# the gateway host when the Podman driver starts, so a bad path or a
# certificate-free bundle fails gateway startup rather than a later sandbox.
# additional_ca_bundle = "/etc/openshell/tls/additional-ca.pem"
# Project a host Workload API Unix socket into the supervisor, or use an
# explicit container-reachable TCP endpoint, for provider token exchange.
# provider_spiffe_workload_api_socket = "/run/spire/agent.sock"
e2e/rust/Cargo.toml
+6
View File
@@ -26,6 +26,7 @@ e2e = []
# when it has resolved an IP.
e2e-host-gateway = ["e2e"]
e2e-local-container-driver = ["e2e"]
e2e-additional-ca = ["e2e", "e2e-host-gateway"]
e2e-docker = ["e2e", "e2e-host-gateway", "e2e-local-container-driver"]
# Selects tests that need a Docker gateway with config_delivery_mode = "push".
e2e-config-push = ["e2e-docker"]
@@ -98,6 +99,11 @@ name = "docker_corporate_proxy"
path = "tests/docker_corporate_proxy.rs"
required-features = ["e2e-docker"]
[[test]]
name = "additional_ca"
path = "tests/additional_ca.rs"
required-features = ["e2e-additional-ca"]
[[test]]
name = "driver_config_volume"
path = "tests/driver_config_volume.rs"
e2e/rust/e2e-docker.sh
+5
View File
@@ -18,6 +18,11 @@ RUN_WITH_GATEWAY_COMMAND="__openshell_run_docker_e2e"
# shellcheck source=e2e/support/conformance.sh
source "${ROOT}/e2e/support/conformance.sh"
if [ "${E2E_TEST}" = "additional_ca" ]; then
export OPENSHELL_E2E_ADDITIONAL_CA=1
E2E_FEATURES="${E2E_FEATURES},e2e-additional-ca"
fi
run_e2e() {
e2e_run_openshell_conformance "Docker"
e2e/rust/e2e-kubernetes.sh
+8
View File
@@ -27,8 +27,16 @@ RUN_WITH_GATEWAY_COMMAND="__openshell_run_kubernetes_e2e"
source "${ROOT}/e2e/support/conformance.sh"
# Unset or empty selects Rust defaults; e2e-conformance-only skips Rust tests.
# Reusable-workflow inputs are exported as an explicitly empty environment
# variable when the caller uses their default. Treat that the same as unset so
# the focused additional-CA row retains the upstream Kubernetes feature set.
E2E_FEATURES="${OPENSHELL_E2E_KUBERNETES_FEATURES:-e2e,e2e-host-gateway,e2e-kubernetes}"
if [ "${OPENSHELL_E2E_KUBE_TEST:-}" = "additional_ca" ]; then
export OPENSHELL_E2E_ADDITIONAL_CA=1
E2E_FEATURES="${E2E_FEATURES},e2e-additional-ca"
fi
# Fixed output path of the `e2e-kubernetes` nextest profile (`.config/nextest.toml`).
JUNIT_XML="${ROOT}/results/e2e-kubernetes.xml"
e2e/rust/e2e-podman.sh
+5
View File
@@ -67,6 +67,11 @@ if [ -n "${E2E_TEST}" ]; then
SELECTED_TESTS=("${E2E_TEST}")
fi
if [ "${E2E_TEST}" = "additional_ca" ]; then
export OPENSHELL_E2E_ADDITIONAL_CA=1
E2E_FEATURES="${E2E_FEATURES},e2e-additional-ca"
fi
if [ "${E2E_TEST}" = "gpu" ] && [ -z "${OPENSHELL_E2E_WORKLOAD_MANIFEST:-}" ] && [ ! -f "${DEFAULT_WORKLOAD_MANIFEST}" ]; then
echo "note: running Podman GPU e2e without a workload manifest; workload validation will log an explicit skip. Build one with 'CONTAINER_ENGINE=podman mise run e2e:workloads:build' or set OPENSHELL_E2E_WORKLOAD_MANIFEST."
fi
e2e/rust/e2e-vm.sh
+58
View File
@@ -52,6 +52,11 @@ CLI_BIN="${OPENSHELL_BIN:-${ROOT}/target/debug/openshell}"
E2E_TEST_OVERRIDE="${OPENSHELL_E2E_VM_TEST:-}"
# Unset or empty selects Rust defaults; e2e-conformance-only skips Rust tests.
E2E_FEATURES="${OPENSHELL_E2E_VM_FEATURES:-e2e-vm}"
ADDITIONAL_CA_MODE="${OPENSHELL_E2E_ADDITIONAL_CA:-0}"
if [ "${E2E_TEST_OVERRIDE}" = "additional_ca" ]; then
ADDITIONAL_CA_MODE=1
E2E_FEATURES="${OPENSHELL_E2E_VM_FEATURES:-e2e-vm,e2e-additional-ca}"
fi
SANDBOX_IMAGE="${OPENSHELL_SANDBOX_IMAGE:-nvcr.io/nvidia/base/ubuntu:24.04}"
# The VM driver places `compute-driver.sock` under `[openshell.drivers.vm].state_dir`.
@@ -169,8 +174,13 @@ s.close()')"
# basename short — see the SUN_LEN comment above.
RUN_STATE_DIR="${STATE_DIR_ROOT}/os-vm-e2e-${HOST_PORT}-$$"
mkdir -p "${RUN_STATE_DIR}"
# Keep every gateway-created artifact, including normalized additional CA
# material, under this invocation's disposable state root. Set this before
# starting either fixture or gateway so no host-global XDG fallback is used.
export XDG_CONFIG_HOME="${RUN_STATE_DIR}/config"
export XDG_DATA_HOME="${RUN_STATE_DIR}/data"
export XDG_STATE_HOME="${RUN_STATE_DIR}/state"
mkdir -p "${XDG_CONFIG_HOME}" "${XDG_DATA_HOME}" "${XDG_STATE_HOME}"
GATEWAY_LOG="$(mktemp /tmp/openshell-gateway-e2e.XXXXXX)"
GATEWAY_PID_FILE="${RUN_STATE_DIR}/gateway.pid"
@@ -183,6 +193,10 @@ GATEWAY_NAME="openshell-e2e-vm-${HOST_PORT}"
DRIVER_PID=""
DRIVER_LOG="${RUN_STATE_DIR}/vm-driver.log"
DRIVER_SOCKET="${RUN_STATE_DIR}/compute-driver.sock"
ADDITIONAL_CA_SERVER_PID=""
ADDITIONAL_CA_SERVER_LOG="${RUN_STATE_DIR}/additional-ca-server.log"
ADDITIONAL_CA_DIR="${RUN_STATE_DIR}/additional-ca"
ADDITIONAL_CA_PORT=""
# ── Cleanup (trap) ───────────────────────────────────────────────────
@@ -207,6 +221,7 @@ cleanup() {
wait "${gateway_pid}" 2>/dev/null || true
fi
e2e_stop_process "${DRIVER_PID}" "external VM compute driver"
e2e_stop_process "${ADDITIONAL_CA_SERVER_PID}" "additional CA HTTPS fixture"
# On failure, keep the VM console log for debugging. We deliberately
# print it instead of leaving it on disk because the state dir gets
@@ -242,6 +257,19 @@ trap cleanup EXIT
# ── Launch the gateway + VM driver ───────────────────────────────────
start_additional_ca_fixture() {
e2e_start_additional_ca_fixture \
"${ADDITIONAL_CA_DIR}" "${ADDITIONAL_CA_SERVER_LOG}" \
ADDITIONAL_CA_SERVER_PID ADDITIONAL_CA_PORT
}
if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then
if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then
echo "ERROR: additional CA e2e requires the gateway-managed VM driver." >&2
exit 2
fi
start_additional_ca_fixture
fi
echo "==> Starting openshell-gateway on 127.0.0.1:${HOST_PORT} (state: ${RUN_STATE_DIR})"
# Pin `driver_dir` to the workspace `target/debug/` so we always pick up
@@ -280,6 +308,15 @@ gateway_id = "${GATEWAY_NAME}"
# Local VM e2e gateways exercise the single-player default: sandbox JWTs
# identify the supervisor and do not expire.
EOF
if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then
cat >>"${GATEWAY_CONFIG}" <<EOF
[openshell.supervisor.network]
additional_ca_bundle = "${ADDITIONAL_CA_DIR}/ca.crt"
EOF
fi
cat >>"${GATEWAY_CONFIG}" <<EOF
[openshell.drivers.vm]
EOF
# A rootless Linux libkrun refuses file creation on read-write bind mounts by
@@ -393,6 +430,27 @@ fi
export OPENSHELL_E2E_DRIVER="vm"
export OPENSHELL_E2E_VM_STATE_DIR="${RUN_STATE_DIR}"
if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then
additional_ca_artifact_dir="${XDG_STATE_HOME}/openshell/network-supervisor"
additional_ca_artifacts=("${additional_ca_artifact_dir}"/additional-ca-*.crt)
if [ "${#additional_ca_artifacts[@]}" -ne 1 ] || [ ! -f "${additional_ca_artifacts[0]}" ]; then
echo "ERROR: expected exactly one content-addressed additional CA artifact under XDG_STATE_HOME" >&2
exit 1
fi
# Both paths exist after gateway initialization; plain `realpath` is
# portable across GNU and BSD/macOS implementations.
state_root="$(realpath "${XDG_STATE_HOME}")"
artifact_path="$(realpath "${additional_ca_artifacts[0]}")"
case "${artifact_path}" in
"${state_root}"/*) ;;
*)
echo "ERROR: normalized additional CA artifact escaped XDG_STATE_HOME" >&2
exit 1
;;
esac
export OPENSHELL_E2E_ADDITIONAL_CA_ARTIFACT="${artifact_path}"
export OPENSHELL_E2E_GATEWAY_CONFIG="${GATEWAY_CONFIG}"
fi
e2e_export_gateway_restart_metadata \
"${GATEWAY_BIN}" \
"${GATEWAY_ARGS_FILE}" \
e2e/rust/tests/additional_ca.rs
+1232
View File
File diff suppressed because it is too large Load Diff
e2e/support/gateway-common.sh
+92
View File
@@ -126,6 +126,98 @@ e2e_pick_port() {
python3 -c 'import socket; s=socket.socket(); s.bind(("",0)); print(s.getsockname()[1]); s.close()'
}
# Generate one ephemeral private destination CA and matching-host HTTPS fixture
# for every compute-driver lane. The caller owns cleanup of the returned PID.
e2e_start_additional_ca_fixture() {
local fixture_dir=$1
local fixture_log=$2
local pid_var=$3
local port_var=$4
local fixture_pid fixture_port
mkdir -p "${fixture_dir}"
(
umask 077
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 \
-out "${fixture_dir}/ca.key" >/dev/null 2>&1
openssl req -x509 -new -key "${fixture_dir}/ca.key" \
-out "${fixture_dir}/ca.crt" -days 1 \
-subj '/CN=OpenShell additional CA e2e' \
-addext 'basicConstraints=critical,CA:TRUE' \
-addext 'keyUsage=critical,keyCertSign,cRLSign' >/dev/null 2>&1
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 \
-out "${fixture_dir}/server.key" >/dev/null 2>&1
openssl req -new -key "${fixture_dir}/server.key" \
-out "${fixture_dir}/server.csr" \
-subj '/CN=host.openshell.internal' >/dev/null 2>&1
cat >"${fixture_dir}/server.ext" <<'EOF'
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:host.openshell.internal
EOF
openssl x509 -req -in "${fixture_dir}/server.csr" \
-CA "${fixture_dir}/ca.crt" -CAkey "${fixture_dir}/ca.key" \
-CAcreateserial -out "${fixture_dir}/server.crt" -days 1 \
-extfile "${fixture_dir}/server.ext" >/dev/null 2>&1
)
fixture_port="$(e2e_pick_port)"
python3 -u - "${fixture_port}" \
"${fixture_dir}/server.crt" "${fixture_dir}/server.key" \
>"${fixture_log}" 2>&1 <<'PY' &
import http.server
import ssl
import sys
port = int(sys.argv[1])
context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
context.load_cert_chain(sys.argv[2], sys.argv[3])
class Server(http.server.ThreadingHTTPServer):
def handle_error(self, request, client_address):
# Hostname-mismatch probes intentionally abort during TLS setup.
pass
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
body = b'{"additional_ca":"trusted"}'
self.send_response(200)
self.send_header('Content-Type', 'application/json')
self.send_header('Content-Length', str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, fmt, *args):
pass
server = Server(('0.0.0.0', port), Handler)
server.socket = context.wrap_socket(server.socket, server_side=True)
print('additional-ca-server-listening', flush=True)
server.serve_forever()
PY
fixture_pid=$!
printf -v "${pid_var}" '%s' "${fixture_pid}"
printf -v "${port_var}" '%s' "${fixture_port}"
local elapsed=0
while [ "${elapsed}" -lt 30 ]; do
if ! kill -0 "${fixture_pid}" 2>/dev/null; then
echo "ERROR: additional CA HTTPS fixture exited before becoming ready" >&2
return 1
fi
if curl -kfsS --max-time 2 "https://127.0.0.1:${fixture_port}/" >/dev/null 2>&1; then
export OPENSHELL_E2E_ADDITIONAL_CA_PORT="${fixture_port}"
export OPENSHELL_E2E_ADDITIONAL_CA_CERT="${fixture_dir}/ca.crt"
return 0
fi
sleep 1
elapsed=$((elapsed + 1))
done
echo "ERROR: additional CA HTTPS fixture did not become ready" >&2
return 1
}
e2e_generate_pki() {
local gateway_bin=$1
local pki_dir=$2
e2e/with-docker-gateway.sh
+37
View File
@@ -126,6 +126,11 @@ DOCKER_NETWORK_MANAGED=0
GPU_MODE="${OPENSHELL_E2E_DOCKER_GPU:-0}"
OIDC_MODE="${OPENSHELL_E2E_OIDC_GATEWAY:-0}"
OIDC_ISSUER="${OPENSHELL_E2E_OIDC_ISSUER:-}"
ADDITIONAL_CA_MODE="${OPENSHELL_E2E_ADDITIONAL_CA:-0}"
ADDITIONAL_CA_SERVER_PID=""
ADDITIONAL_CA_SERVER_LOG="${WORKDIR}/additional-ca-server.log"
ADDITIONAL_CA_DIR="${WORKDIR}/additional-ca"
ADDITIONAL_CA_PORT=""
if [ "${OIDC_MODE}" = "1" ] && [ -z "${OIDC_ISSUER}" ]; then
echo "ERROR: OPENSHELL_E2E_OIDC_ISSUER is required when OPENSHELL_E2E_OIDC_GATEWAY=1" >&2
@@ -145,6 +150,7 @@ cleanup() {
e2e_stop_gateway "${GATEWAY_PID}" "${GATEWAY_PID_FILE}"
e2e_stop_process "${DRIVER_PID}" "external Docker compute driver"
e2e_stop_process "${ADDITIONAL_CA_SERVER_PID}" "additional CA HTTPS fixture"
if [ "${exit_code}" -ne 0 ] \
&& [ -n "${E2E_NAMESPACE}" ] \
@@ -198,6 +204,11 @@ cleanup() {
cat "${DRIVER_LOG}" || true
echo "=== end external Docker compute driver log ==="
fi
if [ "${exit_code}" -ne 0 ] && [ -f "${ADDITIONAL_CA_SERVER_LOG}" ]; then
echo "=== additional CA HTTPS fixture log ==="
cat "${ADDITIONAL_CA_SERVER_LOG}" || true
echo "=== end additional CA HTTPS fixture log ==="
fi
rm -rf "${WORKDIR}" 2>/dev/null || true
}
@@ -268,6 +279,10 @@ connect_current_container_to_docker_network() {
}
if [ -n "${OPENSHELL_GATEWAY_ENDPOINT:-}" ]; then
if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then
echo "ERROR: additional CA e2e requires the wrapper-managed Docker gateway." >&2
exit 2
fi
case "${OPENSHELL_GATEWAY_ENDPOINT}" in
http://*) ;;
https://*)
@@ -544,6 +559,19 @@ if ! ensure_sandbox_image_available "${SANDBOX_IMAGE}"; then
exit 2
fi
start_additional_ca_fixture() {
e2e_start_additional_ca_fixture \
"${ADDITIONAL_CA_DIR}" "${ADDITIONAL_CA_SERVER_LOG}" \
ADDITIONAL_CA_SERVER_PID ADDITIONAL_CA_PORT
}
if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then
if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then
echo "ERROR: additional CA e2e requires the in-process Docker driver." >&2
exit 2
fi
start_additional_ca_fixture
fi
HOST_PORT=$(e2e_pick_port)
HEALTH_PORT=$(e2e_pick_port)
STATE_DIR="${XDG_STATE_HOME}"
@@ -561,6 +589,10 @@ export OPENSHELL_E2E_DOCKER_NETWORK_NAME="${DOCKER_NETWORK_NAME}"
export OPENSHELL_E2E_NETWORK_NAME="${DOCKER_NETWORK_NAME}"
export OPENSHELL_E2E_SANDBOX_NAMESPACE="${E2E_NAMESPACE}"
export OPENSHELL_E2E_DRIVER="docker"
if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then
export OPENSHELL_E2E_ADDITIONAL_CA_ARTIFACT="${XDG_STATE_HOME}/openshell/network-supervisor"
export OPENSHELL_E2E_GATEWAY_CONFIG="${STATE_DIR}/gateway.toml"
fi
if connect_current_container_to_docker_network "${DOCKER_NETWORK_NAME}"; then
echo "Connected CI job container to Docker network ${DOCKER_NETWORK_NAME} (${GATEWAY_HOST_ALIAS_IP})."
# Container jobs use the host Docker daemon. The host-networked supervisor
@@ -613,6 +645,11 @@ GATEWAY_CONFIG="${STATE_DIR}/gateway.toml"
e2e_write_gateway_oidc_config "${OPENSHELL_OIDC_ISSUER}"
fi
fi
if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then
printf '[openshell.supervisor.network]\n'
printf 'additional_ca_bundle = %s\n\n' \
"$(toml_string "${ADDITIONAL_CA_DIR}/ca.crt")"
fi
printf '[openshell.drivers.docker]\n'
if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then
printf 'socket_path = %s\n' "$(toml_string "${DRIVER_SOCKET}")"
e2e/with-kube-gateway.sh
+53
View File
@@ -128,6 +128,12 @@ OPENSHIFT_ROUTE_HOST=""
# Temp dir holding the client mTLS material extracted from openshell-client-tls
# for the OpenShift Route transport. Removed by cleanup().
OPENSHIFT_PKI_DIR="${WORKDIR}/openshift-pki"
ADDITIONAL_CA_MODE="${OPENSHELL_E2E_ADDITIONAL_CA:-0}"
ADDITIONAL_CA_SOURCE_CONFIG_MAP="openshell-e2e-additional-ca"
ADDITIONAL_CA_SERVER_PID=""
ADDITIONAL_CA_SERVER_LOG="${WORKDIR}/additional-ca-server.log"
ADDITIONAL_CA_DIR="${WORKDIR}/additional-ca"
ADDITIONAL_CA_SOURCE_CREATED=0
# Isolate CLI/SDK gateway metadata from the developer's real config.
export XDG_CONFIG_HOME="${WORKDIR}/config"
@@ -510,6 +516,15 @@ cleanup() {
local exit_code=$?
stop_gateway_portforward
e2e_stop_process "${ADDITIONAL_CA_SERVER_PID}" "additional CA HTTPS fixture"
if [ "${ADDITIONAL_CA_SOURCE_CREATED}" = "1" ] \
&& [ -n "${KUBE_CONTEXT}" ] && [ -n "${NAMESPACE}" ] \
&& command -v kubectl >/dev/null 2>&1; then
kctl -n "${NAMESPACE}" delete configmap "${ADDITIONAL_CA_SOURCE_CONFIG_MAP}" \
--ignore-not-found >/dev/null 2>&1 || true
ADDITIONAL_CA_SOURCE_CREATED=0
fi
if [ "${exit_code}" -ne 0 ] && [ -n "${KUBE_CONTEXT}" ] && [ -n "${NAMESPACE}" ]; then
if command -v kubectl >/dev/null 2>&1 \
@@ -763,6 +778,12 @@ run_scenario() {
export OPENSHELL_E2E_KUBE_CHART="${ROOT}/deploy/helm/openshell"
export OPENSHELL_E2E_KUBECONFIG="${KUBECONFIG:-}"
export OPENSHELL_PROVISION_TIMEOUT="${OPENSHELL_PROVISION_TIMEOUT:-300}"
if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then
export OPENSHELL_E2E_ADDITIONAL_CA_HELM_NAMESPACE="${NAMESPACE}"
export OPENSHELL_E2E_ADDITIONAL_CA_HELM_RELEASE="${RELEASE_NAME}"
export OPENSHELL_E2E_ADDITIONAL_CA_HELM_CHART="${ROOT}/deploy/helm/openshell"
export OPENSHELL_E2E_KUBE_GATEWAY_LOCAL_PORT="${LOCAL_PORT:-}"
fi
e2e_import_example_provider_profiles \
"${OPENSHELL_BIN:-${ROOT}/target/debug/openshell}" "${ROOT}" || return 1
@@ -1224,6 +1245,29 @@ if kctl api-resources --api-group=route.openshift.io --no-headers 2>/dev/null |
fi
fi
start_additional_ca_fixture() {
[ "${ADDITIONAL_CA_MODE}" = "1" ] || return 0
if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then
echo "ERROR: Kubernetes additional CA e2e requires the in-process Kubernetes driver." >&2
return 1
fi
if [ -z "${HOST_GATEWAY_IP}" ]; then
echo "ERROR: Kubernetes additional CA e2e requires a host gateway IP for host.openshell.internal." >&2
return 1
fi
e2e_start_additional_ca_fixture \
"${ADDITIONAL_CA_DIR}" "${ADDITIONAL_CA_SERVER_LOG}" \
ADDITIONAL_CA_SERVER_PID ADDITIONAL_CA_PORT
kctl create namespace "${NAMESPACE}" --dry-run=client -o yaml | kctl apply -f -
kctl -n "${NAMESPACE}" create configmap "${ADDITIONAL_CA_SOURCE_CONFIG_MAP}" \
--from-file="ca.crt=${ADDITIONAL_CA_DIR}/ca.crt" \
--dry-run=client -o yaml | kctl apply -f -
ADDITIONAL_CA_SOURCE_CREATED=1
}
start_additional_ca_fixture
ACTIVE_CREDENTIAL_DRIVER="${OPENSHELL_E2E_CREDENTIAL_DRIVER:-kubernetes-secrets}"
if [ "${OPENSHELL_E2E_CREDENTIAL_DRIVERS:-0}" = "1" ] \
&& [ "${ACTIVE_CREDENTIAL_DRIVER}" = "vault" ]; then
@@ -1234,6 +1278,9 @@ helm_extra_args=()
helm_post_renderer_args=()
helm_extra_args+=(--set "server.telemetryEnabled=${OPENSHELL_TELEMETRY_ENABLED}")
helm_extra_args+=(--set supervisor.sandboxRuntime.networkPolicyEnforced=true)
if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then
helm_extra_args+=(--set "supervisor.network.additionalCaConfigMapName=${ADDITIONAL_CA_SOURCE_CONFIG_MAP}")
fi
if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then
if [ "${OPENSHELL_E2E_KUBE_BUILD_IMAGES}" != "1" ]; then
echo "ERROR: external Kubernetes driver e2e requires OPENSHELL_E2E_KUBE_BUILD_IMAGES=1." >&2
@@ -1527,6 +1574,12 @@ else
export OPENSHELL_E2E_KUBE_CHART="${ROOT}/deploy/helm/openshell"
export OPENSHELL_E2E_KUBECONFIG="${KUBECONFIG:-}"
export OPENSHELL_PROVISION_TIMEOUT="${OPENSHELL_PROVISION_TIMEOUT:-300}"
if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then
export OPENSHELL_E2E_ADDITIONAL_CA_HELM_NAMESPACE="${NAMESPACE}"
export OPENSHELL_E2E_ADDITIONAL_CA_HELM_RELEASE="${RELEASE_NAME}"
export OPENSHELL_E2E_ADDITIONAL_CA_HELM_CHART="${ROOT}/deploy/helm/openshell"
export OPENSHELL_E2E_KUBE_GATEWAY_LOCAL_PORT="${LOCAL_PORT:-}"
fi
e2e_import_example_provider_profiles \
"${OPENSHELL_BIN:-${ROOT}/target/debug/openshell}" "${ROOT}" || exit 1
e2e/with-podman-gateway.sh
+46
View File
@@ -131,6 +131,11 @@ PODMAN_SOCKET=""
GPU_MODE="${OPENSHELL_E2E_PODMAN_GPU:-0}"
OIDC_MODE="${OPENSHELL_E2E_OIDC_GATEWAY:-0}"
OIDC_ISSUER="${OPENSHELL_E2E_OIDC_ISSUER:-}"
ADDITIONAL_CA_MODE="${OPENSHELL_E2E_ADDITIONAL_CA:-0}"
ADDITIONAL_CA_SERVER_PID=""
ADDITIONAL_CA_SERVER_LOG="${WORKDIR}/additional-ca-server.log"
ADDITIONAL_CA_DIR="${WORKDIR}/additional-ca"
ADDITIONAL_CA_PORT=""
if [ "${OIDC_MODE}" = "1" ] && [ -z "${OIDC_ISSUER}" ]; then
echo "ERROR: OPENSHELL_E2E_OIDC_ISSUER is required when OPENSHELL_E2E_OIDC_GATEWAY=1" >&2
@@ -145,6 +150,7 @@ cleanup() {
e2e_stop_gateway "${GATEWAY_PID}" "${GATEWAY_PID_FILE}"
e2e_stop_process "${DRIVER_PID}" "external Podman compute driver"
e2e_stop_process "${ADDITIONAL_CA_SERVER_PID}" "additional CA HTTPS fixture"
local sandbox_ids=""
if command -v podman >/dev/null 2>&1; then
@@ -215,6 +221,11 @@ cleanup() {
cat "${PODMAN_SERVICE_LOG}" || true
echo "=== end podman service log ==="
fi
if [ "${exit_code}" -ne 0 ] && [ -f "${ADDITIONAL_CA_SERVER_LOG}" ]; then
echo "=== additional CA HTTPS fixture log ==="
cat "${ADDITIONAL_CA_SERVER_LOG}" || true
echo "=== end additional CA HTTPS fixture log ==="
fi
if [ -n "${PODMAN_SERVICE_PID}" ]; then
kill "${PODMAN_SERVICE_PID}" >/dev/null 2>&1 || true
@@ -441,6 +452,10 @@ ensure_podman_sandbox_runtime_image() {
}
if [ -n "${OPENSHELL_GATEWAY_ENDPOINT:-}" ]; then
if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then
echo "ERROR: additional CA e2e requires the wrapper-managed Podman gateway." >&2
exit 2
fi
case "${OPENSHELL_GATEWAY_ENDPOINT}" in
http://*) ;;
https://*)
@@ -515,6 +530,19 @@ PKI_DIR="${WORKDIR}/pki"
e2e_generate_pki "${GATEWAY_BIN}" "${PKI_DIR}" "host.containers.internal"
export OPENSHELL_E2E_GATEWAY_CA_CERT="${PKI_DIR}/ca.crt"
start_additional_ca_fixture() {
e2e_start_additional_ca_fixture \
"${ADDITIONAL_CA_DIR}" "${ADDITIONAL_CA_SERVER_LOG}" \
ADDITIONAL_CA_SERVER_PID ADDITIONAL_CA_PORT
}
if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then
if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then
echo "ERROR: additional CA e2e requires the in-process Podman driver." >&2
exit 2
fi
start_additional_ca_fixture
fi
HOST_PORT=$(e2e_pick_port)
HEALTH_PORT=$(e2e_pick_port)
PRIMARY_BIND_IP="127.0.0.1"
@@ -532,6 +560,10 @@ ensure_e2e_podman_network "${PODMAN_NETWORK_NAME}"
export OPENSHELL_E2E_DRIVER="podman"
export OPENSHELL_E2E_NETWORK_NAME="${PODMAN_NETWORK_NAME}"
export OPENSHELL_E2E_SANDBOX_NAMESPACE="${E2E_NAMESPACE}"
if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then
export OPENSHELL_E2E_ADDITIONAL_CA_ARTIFACT="${XDG_STATE_HOME}/openshell/network-supervisor"
export OPENSHELL_E2E_GATEWAY_CONFIG="${STATE_DIR}/gateway.toml"
fi
echo "Starting openshell-gateway on port ${HOST_PORT} (namespace: ${E2E_NAMESPACE})..."
e2e_generate_gateway_jwt "${JWT_DIR}"
@@ -555,6 +587,20 @@ e2e_write_podman_gateway_config \
"${OPENSHELL_PODMAN_SOCKET:-}" \
"${OIDC_MODE}" \
"${OPENSHELL_OIDC_ISSUER:-}"
if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then
CONFIG_WITH_ADDITIONAL_CA="${GATEWAY_CONFIG}.additional-ca"
awk -v ca_path="$(e2e_podman_toml_string "${ADDITIONAL_CA_DIR}/ca.crt")" '
!inserted && $0 == "[openshell.drivers.podman]" {
print "[openshell.supervisor.network]"
print "additional_ca_bundle = " ca_path
print ""
inserted = 1
}
{ print }
END { if (!inserted) exit 42 }
' "${GATEWAY_CONFIG}" >"${CONFIG_WITH_ADDITIONAL_CA}"
mv "${CONFIG_WITH_ADDITIONAL_CA}" "${GATEWAY_CONFIG}"
fi
EXTERNAL_DRIVER_GRPC_ENDPOINT="https://127.0.0.1:${HOST_PORT}"
EXTERNAL_DRIVER_HEALTH_CHECK_INTERVAL_SECS=10
EXTERNAL_DRIVER_ENABLE_BIND_MOUNTS=true
examples/governance-interceptor/Cargo.lock
Generated
+1
View File
@@ -1860,6 +1860,7 @@ dependencies = [
"base64 0.22.1",
"glob",
"ipnet",
"libc",
"miette",
"nix",
"openshell-crypto",
examples/isolation-backend-v8/Cargo.lock
Generated
+1
View File
@@ -2129,6 +2129,7 @@ dependencies = [
"glob",
"ipnet",
"jsonwebtoken",
"libc",
"miette",
"nix",
"openshell-crypto",
examples/supervisor-middleware-content-guard/Cargo.lock
Generated
+1
View File
@@ -2014,6 +2014,7 @@ dependencies = [
"base64 0.22.1",
"glob",
"ipnet",
"libc",
"miette",
"nix",
"openshell-crypto",
proto/compute_driver.proto
+6
View File
@@ -480,6 +480,12 @@ message StartSandboxRequest {
bool resume = 7;
// Forwarded to ResumeBoundaryRequest.restart_if_restore_fails.
bool restart_if_restore_fails = 8;
// Durable provisioning snapshot recovered by the gateway from its persisted
// Sandbox record. Local drivers use this only to reconcile a stopped
// workload/supervisor pair without trusting mutable runtime inspection
// output. Older gateways omit the field, preserving the original start
// contract on the wire.
optional DriverSandbox sandbox = 9;
}
message SuspendSandboxRequest {
rfc/0003-gateway-configuration/README.md
+9 -2
View File
@@ -96,6 +96,11 @@ disable_tls = false
# Gateway-owned CA injected into the selected local driver for supervisor TLS.
guest_tls_ca = "/etc/openshell/certs/ca.pem"
# Shared sandbox destination trust. These gateway-local paths are read and
# normalized at startup; the resulting material is delivered by in-tree drivers.
[openshell.supervisor.network]
additional_ca_bundle = "/etc/openshell/certs/private-destination-ca.pem"
[openshell.gateway.tls]
cert_path = "/etc/openshell/certs/gateway.pem"
key_path = "/etc/openshell/certs/gateway-key.pem"
@@ -172,8 +177,9 @@ Driver authors define and own their config schema. Adding a new driver does not
Field-level merge rules:
1. **`[openshell.gateway]`** populates `openshell_core::Config` (including the nested `[openshell.gateway.tls]` and `[openshell.gateway.oidc]` tables, which map to `TlsConfig` and `OidcConfig` respectively).
2. **`[openshell.drivers.<name>]`** is propagated to the driver crate, which deserializes it into its own struct. Driver schemas evolve independently of the gateway's core `Config`.
3. **CLI / env** override any value set by steps 1–2, field by field. The override check uses clap's `ValueSource` — a value is applied from the file only when the corresponding flag was not supplied via the command line or environment.
2. **`[openshell.supervisor.network]`** holds shared network-supervisor settings. `additional_ca_bundle` names a single gateway-local certificate-only PEM file. Gateway startup strictly normalizes it before an in-tree compute driver receives the material. The setting augments sandbox destination trust and does not configure proxy, listener, OIDC, or gateway callback trust.
3. **`[openshell.drivers.<name>]`** is propagated to the driver crate, which deserializes it into its own struct. Driver schemas evolve independently of the gateway's core `Config`.
4. **CLI / env** override any value set by steps 1–3, field by field. The override check uses clap's `ValueSource` — a value is applied from the file only when the corresponding flag was not supplied via the command line or environment.
`bind_address`, `health_bind_address`, and `metrics_bind_address` are stored as `SocketAddr` (IP + port). The CLI exposes them as a single `--bind-address` IP plus `--port`, `--health-port`, and `--metrics-port`; CLI overrides apply to the matching part of the parsed `SocketAddr`.
@@ -203,6 +209,7 @@ The following cross-field validations are applied after merging file + env + CLI
- Gateway listener TLS requires `cert_path` and `key_path`; `client_ca_path` is required only for listener client-certificate verification. TLS-enabled Docker, Podman, and VM drivers also require a complete gateway-owned guest CA, certificate, and key bundle. Kubernetes projects guest TLS through a Secret instead.
- `database_url` must be non-empty after merging env + CLI — every supported driver requires it. The field is not accepted from the file (see Secrets above).
- `compute_driver` selects exactly one driver. When omitted, the gateway falls back to auto-detection. A custom driver requires a named table with `socket_path`, unless startup supplies an explicit socket override. The legacy `compute_drivers` list is rejected.
- `additional_ca_bundle` must name a readable, non-empty certificate-only PEM file with at least one usable certificate. Private keys, unrelated PEM items, DER-only input, inline PEM, and URLs are rejected. Configured material is unsupported with remote or custom compute drivers.
### Schema compatibility
skills/debug-openshell-cluster/SKILL.md
+91 -5
View File
@@ -427,13 +427,13 @@ Common findings:
probe and the companion supervisor's private health check. Do not add
capabilities, attach a workload network, or disable the runtime seccomp
profile. There is no sandbox nftables or nested-network setup to repair.
- Gateway fails to start with an `additional_ca_bundle` error: the Podman
driver validates the setting at startup. The file must exist on the gateway
host, be non-empty, and contain at least one PEM certificate.
- Gateway fails to start with an `additional_ca_bundle` error: gateway startup
validates the configured single bundle. The file must exist on the gateway
host, be a regular non-symlink file, and contain at least one PEM certificate.
- Inspected HTTPS to a private-CA destination fails (for example
`curl: (52) Empty reply from server`), or the supervisor exits at startup,
with `additional_ca_bundle` set: the supervisor container needs the CA, not
just the workload. Check
with a global `additional_ca_bundle` configured: the supervisor container
needs the staged CA, not just the workload. Check
`podman inspect <sandbox>-supervisor --format '{{json .Mounts}}'` for a
read-only bind of the bundle at `/etc/openshell/tls/additional/ca-bundle.pem`,
and `--format '{{json .Config.Cmd}}'` for `--additional-ca-bundle` with that
@@ -523,6 +523,92 @@ kubectl -n openshell logs <gateway-pod> -c openshell-gateway --tail=200
`server.telemetryEnabled` renders `OPENSHELL_TELEMETRY_ENABLED` on the gateway
pod, and the gateway propagates the effective value to sandbox supervisors.
#### Additional destination CA ConfigMap
When `supervisor.network.additionalCaConfigMapName` is set, Helm must mount the
**source** operator ConfigMap's `ca.crt` key into the gateway at
`/etc/openshell-tls/additional-ca-source/ca.crt`. The rendered
`gateway.toml` must contain the global (not Kubernetes-driver-local) setting:
```toml
[openshell.supervisor.network]
additional_ca_bundle = "/etc/openshell-tls/additional-ca-source/ca.crt"
```
Inspect the source, render, and live gateway mount before inspecting sandbox
pods:
```bash
helm -n openshell get values openshell | grep -A3 additionalCaConfigMapName
kubectl -n openshell get configmap <source-configmap> -o jsonpath='{.data.ca\.crt}' >/dev/null
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -A2 '^\[openshell.supervisor.network\]'
kubectl -n openshell get pod <gateway-pod> -o jsonpath='{.spec.volumes[?(@.name=="additional-ca-source")]}{"\n"}'
```
At startup, the gateway normalizes the source once and stages a
content-addressed, read-only artifact below
`/var/lib/openshell-state/openshell/network-supervisor/`. The gateway workload
sets `XDG_STATE_HOME=/var/lib/openshell-state` and mounts the writable
`gateway-state` volume there. Check that wiring before debugging the driver:
```bash
kubectl -n openshell get pod <gateway-pod> -o jsonpath='{.spec.containers[0].env[?(@.name=="XDG_STATE_HOME")]}{"\n"}'
kubectl -n openshell get pod <gateway-pod> -o jsonpath='{.spec.containers[0].volumeMounts[?(@.name=="gateway-state")]}{"\n"}'
kubectl -n openshell get pod <gateway-pod> -o jsonpath='{.spec.volumes[?(@.name=="gateway-state")]}{"\n"}'
```
At sandbox creation or restart, the Kubernetes driver creates or verifies an
immutable, content-addressed ConfigMap in the target sandbox namespace. Its
name binds the effective gateway identity and normalized trust digest. The
managed ConfigMap must have `openshell.ai/managed-by=openshell`, the matching
`openshell.ai/gateway-id` label, and an
`openshell.ai/additional-ca-generation` annotation. Different trust
generations intentionally coexist so running sandboxes retain their startup
trust. The supervisor Pod mounts the ConfigMap read-only at
`/etc/openshell/tls/additional/ca-bundle.pem` and receives
`--additional-ca-bundle` for that path; the immutable flag plus the driver's
per-creation digest and exact-byte validation close deletion/recreation races
after the API check.
Kubernetes RBAC cannot restrict `create` by `resourceNames` or restrict `get`
by a name prefix. When this feature is enabled, the chart grants the gateway
service account ConfigMap `get`/`create` in the shared sandbox namespace. In
managed/operator modes, the existing ClusterRoleBinding makes those permissions
cluster-wide. It does not grant ConfigMap `list`, `watch`, `patch`, `update`, or
`delete`. Prefer a dedicated gateway service account and namespace-scoped
bindings when the read boundary matters. Check the objects and authorization:
```bash
kubectl -n <sandbox-namespace> get configmap \
-l openshell.ai/managed-by=openshell,openshell.ai/gateway-id=<effective-gateway-id> \
--show-labels
kubectl auth can-i get configmaps -n <sandbox-namespace> \
--as system:serviceaccount:openshell:<gateway-service-account>
kubectl auth can-i create configmaps -n <sandbox-namespace> \
--as system:serviceaccount:openshell:<gateway-service-account>
kubectl auth can-i patch configmaps -n <sandbox-namespace> \
--as system:serviceaccount:openshell:<gateway-service-account>
kubectl auth can-i delete configmaps -n <sandbox-namespace> \
--as system:serviceaccount:openshell:<gateway-service-account>
```
A missing, mutable, unowned, wrong-generation, or wrong-data managed ConfigMap
fails closed. Do not relabel or edit it blindly. Investigate the object owner;
remove an invalid object only after confirming no sandbox uses it, then retry
so the driver can recreate the exact generation. Diagnostics deliberately omit
certificate contents; inspect the source object separately rather than copying
PEM into logs or tickets.
Changing source ConfigMap contents does not reload the gateway or a running
supervisor: restart the gateway to reread/normalize it, then recreate or
restart affected sandboxes. Removing the Helm setting means newly created or
recreated sandbox pods have no additional-CA volume/mount, but existing running
pods retain startup trust until restarted. The driver intentionally has no
ConfigMap list/delete permission; unused managed generations can remain. After
all sandboxes are migrated, use an operator identity to verify the generation
annotation and pod references, then explicitly delete unused ConfigMaps in each
applicable namespace. Do not add list/delete RBAC as a cleanup workaround.
When no external credential driver is enabled, the Helm chart uses the
gateway's default encrypted database credential storage. The chart creates a
retained Kubernetes Secret for the shared KEK, injects it into gateway pods, and
tests/testkit/Cargo.lock
Generated
+1
View File
@@ -1888,6 +1888,7 @@ dependencies = [
"chrono",
"glob",
"ipnet",
"libc",
"miette",
"nix",
"openshell-crypto",