refactor(isolation): use backend and runtime terminology

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
This commit is contained in:
Drew Newberry
2026-09-12 14:07:09 -07:00
parent 07441545ba
commit d3d92a7ccc
22 changed files with 297 additions and 313 deletions
+1 -1
View File
@@ -15,7 +15,7 @@ The OpenShell SBOM tooling produces source-tree CycloneDX JSON SBOMs using Syft,
SBOMs are **release artifacts only** -- they are generated on demand and not committed to the repository. Output lands in `deploy/sbom/output/` (gitignored).
Pushed gateway and supervisor images carry an SPDX SBOM and minimal SLSA provenance as OCI attestations. Branch E2E, Release Dev, and Release Tag image binaries embed cargo-auditable metadata, so their image SBOMs include linked Rust crates.
Pushed gateway, sandbox, and supervisor images carry an SPDX SBOM and minimal SLSA provenance as OCI attestations. Branch E2E, Release Dev, and Release Tag image binaries embed cargo-auditable metadata, so their image SBOMs include linked Rust crates.
## Prerequisites
+1 -1
View File
@@ -92,7 +92,7 @@ Note `install.sh` is pulled from `raw.githubusercontent.com/NVIDIA/OpenShell/${h
`Release Dev` publishes two chart versions for every dev build (see `.github/actions/release-helm-oci/action.yml:89-102`):
- `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` — floating, overwritten on every main push.
- `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev.<sha>` — immutable, `appVersion` set to the same SHA so it pulls `ghcr.io/nvidia/openshell/gateway:<sha>` and `:supervisor:<sha>`.
- `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev.<sha>` — immutable, `appVersion` set to the same SHA so it pulls the matching `gateway`, `sandbox`, and `supervisor` images.
To smoke-test the chart for a specific dev build, dispatch `Release Dev` on the branch first, then run the kind canary steps locally pointed at the SHA-pinned chart (see "Local kind reproduction" below). The release-canary workflow itself does not currently expose `chart_version` / `image_tag` inputs.
+5 -1
View File
@@ -38,7 +38,11 @@ These pipelines connect skills into end-to-end workflows. Individual skill files
| `crates/openshell-conformance/` | CLI conformance library | Reusable driver-agnostic scenarios and command runner |
| `crates/openshell-conformance-cli/` | Conformance CLI | Distributable `list` and `run` entrypoint for gateway conformance |
| `crates/openshell-server/` | Gateway server | Control-plane API, sandbox lifecycle, auth boundary |
| `crates/openshell-sandbox/` | Sandbox runtime | Container supervision, policy-enforced egress routing |
| `crates/openshell-sandbox/` | Sandbox runtime | Capability-free workload launcher, process identity, and seccomp-mediated I/O |
| `crates/openshell-supervisor/` | Supervisor runtime | Gateway session, policy evaluation, credentials, and upstream networking |
| `crates/openshell-binary-identity/` | Binary identity | Shared trusted procfs executable identity resolution for isolation backends |
| `crates/openshell-isolation-interface/` | Isolation backend interface | RFC 0012 `IsolationBackend` trait and types; the supervisor-facing runtime contract |
| `crates/openshell-sandbox-backend/` | OpenShell sandbox backend | `OpenShellRuntimeBackend` and the authenticated OpenShell Sandbox Protocol shared with `openshell-sandbox` |
| `crates/openshell-policy/` | Policy engine | Filesystem, network, and process constraints |
| `crates/openshell-bootstrap/` | Gateway metadata | Gateway registration metadata, auth token storage, mTLS bundle storage |
| `crates/openshell-gateway-interceptors/` | Gateway interceptors | Intercepts and transforms configured gRPC requests at the gateway routing boundary |
+3 -3
View File
@@ -359,9 +359,9 @@ and explicit `/sandbox` values select `/sandbox`; other paths must already
exist without symlink or reserved-mount collisions and must be usable by the
resolved identity. Kubernetes and VM use `/sandbox`.
Kubernetes uses only the proxy-pod topology. The driver creates the empty-egress
workload fence before a suspended Sandbox CR, then provisions split immutable
bootstrap Secrets, the boundary Service, and the supervisor Deployment. A
The Kubernetes driver creates the empty-egress workload fence before a
suspended Sandbox CR, then provisions split immutable bootstrap Secrets, the
private runtime Service, and the supervisor Deployment. A
non-root init container stages `openshell-sandbox` and one-use bootstrap files
into memory volumes. The workload Pod never mounts supervisor or gateway
credentials. The driver removes its scheduling gate only after the companions
+1 -1
View File
@@ -883,7 +883,7 @@ system entry instead of pretending to delete package-manager owned state.
- Compute runtimes own the mechanics of starting workloads and injecting
callback configuration. Local Docker, Podman, and VM callback endpoints can
be derived from their fixed host aliases. Kubernetes requires an explicit
endpoint from deployment topology; Helm renders it from the gateway Service
endpoint from driver placement; Helm renders it from the gateway Service
name and namespace rather than inferring it from sandbox placement.
- Docker-backed local gateways use Docker's `host-gateway` callback alias on
macOS and Docker Desktop-style runtimes. They request IPv4 loopback callback
+1 -1
View File
@@ -126,7 +126,7 @@ The sandbox reserves `SIGUSR2` with a non-restarting no-op handler for these
broker threads; startup rejects a conflicting handler. This signal disposition
is process-global kernel state, while registrations and cancellation state are
owned by the broker. Workload exec resets the caught handler to its default.
This topology requires Linux 6.2 or newer for Landlock ABI v3 and treats
This sandbox runtime requires Linux 6.2 or newer for Landlock ABI v3 and treats
`SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV` as mandatory so cancelled
notifications cannot race task-memory writes.
+12 -12
View File
@@ -7,10 +7,10 @@ use std::path::{Path, PathBuf};
use crate::proto::compute::v1::DriverSandbox;
/// Built-in sandbox network topologies used to derive a callback endpoint
/// Built-in sandbox network callback routes used to derive a callback endpoint
/// when an operator does not configure a per-driver `grpc_endpoint` override.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum GatewayCallbackTopology {
pub enum GatewayCallbackRoute {
/// A Docker container reaches the host through Docker's gateway alias.
Docker,
/// A Podman container reaches the host through Podman's gateway alias.
@@ -19,41 +19,41 @@ pub enum GatewayCallbackTopology {
Vm,
}
/// Build the endpoint a sandbox uses to call its gateway for a known topology.
/// Build the endpoint a sandbox uses to call its gateway for a known route.
///
/// The result is deliberately derived by the gateway rather than baked into
/// individual driver defaults. A configured `grpc_endpoint` remains an
/// operator override for remote or non-standard deployments.
#[must_use]
pub fn gateway_callback_endpoint(
topology: GatewayCallbackTopology,
route: GatewayCallbackRoute,
gateway_port: u16,
gateway_tls_enabled: bool,
) -> String {
let scheme = if gateway_tls_enabled { "https" } else { "http" };
let host = match topology {
GatewayCallbackTopology::Docker | GatewayCallbackTopology::Vm => "host.openshell.internal",
GatewayCallbackTopology::Podman => "host.containers.internal",
let host = match route {
GatewayCallbackRoute::Docker | GatewayCallbackRoute::Vm => "host.openshell.internal",
GatewayCallbackRoute::Podman => "host.containers.internal",
};
format!("{scheme}://{host}:{gateway_port}")
}
#[cfg(test)]
mod callback_endpoint_tests {
use super::{GatewayCallbackTopology, gateway_callback_endpoint};
use super::{GatewayCallbackRoute, gateway_callback_endpoint};
#[test]
fn derives_endpoint_for_each_builtin_topology() {
fn derives_endpoint_for_each_builtin_route() {
assert_eq!(
gateway_callback_endpoint(GatewayCallbackTopology::Docker, 17670, false),
gateway_callback_endpoint(GatewayCallbackRoute::Docker, 17670, false),
"http://host.openshell.internal:17670"
);
assert_eq!(
gateway_callback_endpoint(GatewayCallbackTopology::Podman, 17670, true),
gateway_callback_endpoint(GatewayCallbackRoute::Podman, 17670, true),
"https://host.containers.internal:17670"
);
assert_eq!(
gateway_callback_endpoint(GatewayCallbackTopology::Vm, 17670, true),
gateway_callback_endpoint(GatewayCallbackRoute::Vm, 17670, true),
"https://host.openshell.internal:17670"
);
}
+2 -2
View File
@@ -25,7 +25,7 @@ use openshell_core::config::DEFAULT_STOP_TIMEOUT_SECS;
use openshell_core::driver_mounts;
use openshell_core::driver_utils::{
CONDITION_EXITED, CONDITION_RUNTIME_RESTART, CONDITION_WORKSPACE_VALIDATION_FAILED,
GatewayCallbackTopology, LABEL_MANAGED_BY, LABEL_MANAGED_BY_VALUE, LABEL_SANDBOX_ID,
GatewayCallbackRoute, LABEL_MANAGED_BY, LABEL_MANAGED_BY_VALUE, LABEL_SANDBOX_ID,
LABEL_SANDBOX_NAME, LABEL_SANDBOX_NAMESPACE, LABEL_SANDBOX_WORKSPACE,
SUPERVISOR_EXIT_WORKSPACE_VALIDATION_FAILED, SUPERVISOR_IMAGE_BINARY_PATH,
extract_first_tar_entry, gateway_callback_endpoint, supervisor_image_should_refresh,
@@ -589,7 +589,7 @@ impl DockerComputeDriver {
let mut docker_config = docker_config.clone();
if docker_config.grpc_endpoint.trim().is_empty() {
docker_config.grpc_endpoint = gateway_callback_endpoint(
GatewayCallbackTopology::Docker,
GatewayCallbackRoute::Docker,
gateway_port,
docker_guest_tls_configured(&docker_config),
);
+2 -2
View File
@@ -13,7 +13,7 @@ use crate::watcher::{
use openshell_core::ComputeDriverError;
use openshell_core::config::CDI_GPU_DEVICE_ALL;
use openshell_core::driver_utils::{
GatewayCallbackTopology, SUPERVISOR_IMAGE_BINARY_PATH, extract_first_tar_entry,
GatewayCallbackRoute, SUPERVISOR_IMAGE_BINARY_PATH, extract_first_tar_entry,
gateway_callback_endpoint, supervisor_image_should_refresh, temp_extract_container_name,
validate_linux_elf_binary, write_cache_binary_atomic,
};
@@ -440,7 +440,7 @@ impl PodmanComputeDriver {
// topology needs the Podman bridge gateway address.
if config.grpc_endpoint.is_empty() {
config.grpc_endpoint = gateway_callback_endpoint(
GatewayCallbackTopology::Podman,
GatewayCallbackRoute::Podman,
config.gateway_port,
config.tls_enabled(),
);
@@ -10,7 +10,7 @@
//! chain of boxed states:
//!
//! ```text
//! attach topology + sandbox context -> Bound -> confirm -> Ready
//! attach backend descriptor + sandbox context -> Bound -> confirm -> Ready
//! -> start_agent -> Running
//! ```
//!
@@ -27,8 +27,8 @@
//! [`PendingTcpOpen`], resolved for that exact socket and process
//! generation; an unresolved identity denies the open.
//!
//! The contract is transport-neutral. Concrete topology implementations keep
//! their placement and coordination details behind these interfaces.
//! The contract is transport-neutral. Compute drivers keep runtime placement
//! and coordination details behind these interfaces.
use std::collections::{BTreeMap, HashMap};
use std::fmt;
@@ -154,14 +154,14 @@ impl std::error::Error for ResolveError {}
// Descriptor and registry
// ============================================================================
/// The common topology descriptor envelope.
/// The common isolation backend descriptor envelope.
///
/// The compute driver supplies one for every admitted topology. The opaque
/// The compute driver supplies one for the selected isolation backend. The opaque
/// payload identifies an existing resource or carries the trusted prepared
/// inputs the backend needs to establish one during `attach`; its protection
/// and resource lifecycle remain topology-specific.
/// and resource lifecycle remain owned by the compute driver.
#[derive(Debug, Clone)]
pub struct TopologyDescriptor {
pub struct BackendDescriptor {
/// The backend the supervisor must instantiate.
pub backend_name: String,
/// Backend-specific attachment data.
@@ -174,11 +174,11 @@ pub struct TopologyDescriptor {
/// unverified descriptor cannot reach a backend. The type does not imply that
/// the opaque payload has been validated: the backend validates the payload and
/// atomically binds it to the sandbox context during `attach`.
pub struct VerifiedTopologyDescriptor {
descriptor: TopologyDescriptor,
pub struct VerifiedBackendDescriptor {
descriptor: BackendDescriptor,
}
impl VerifiedTopologyDescriptor {
impl VerifiedBackendDescriptor {
/// The verified backend name.
#[must_use]
pub fn backend_name(&self) -> &str {
@@ -309,9 +309,9 @@ impl BackendRegistry {
/// registered for the admitted name.
pub fn resolve(
&self,
descriptor: TopologyDescriptor,
descriptor: BackendDescriptor,
admitted_backend_name: &str,
) -> Result<(Arc<dyn IsolationBackend>, VerifiedTopologyDescriptor), BackendError> {
) -> Result<(Arc<dyn IsolationBackend>, VerifiedBackendDescriptor), BackendError> {
if descriptor.backend_name != admitted_backend_name {
return Err(BackendError::Descriptor(format!(
"descriptor backend {:?} does not match admitted backend {admitted_backend_name:?}",
@@ -330,7 +330,7 @@ impl BackendRegistry {
descriptor.backend_name
)));
}
Ok((backend, VerifiedTopologyDescriptor { descriptor }))
Ok((backend, VerifiedBackendDescriptor { descriptor }))
}
}
@@ -349,7 +349,7 @@ pub trait IsolationBackend: Send + Sync {
/// external orchestrator that supplied the descriptor.
async fn attach(
&self,
descriptor: VerifiedTopologyDescriptor,
descriptor: VerifiedBackendDescriptor,
sandbox: SandboxContext,
) -> Result<Box<dyn BoundBoundary>, BackendError>;
}
@@ -358,7 +358,7 @@ pub trait IsolationBackend: Send + Sync {
// Lifecycle states
// ============================================================================
/// Bound: the topology descriptor and trusted sandbox context are bound to the
/// Bound: the backend descriptor and trusted sandbox context are bound to the
/// same resource, and the mediation source is available. No untrusted workload
/// code is running.
#[async_trait]
@@ -459,38 +459,29 @@ pub enum DriverFenceEvidence {
impl DriverFenceEvidence {
#[must_use]
pub const fn backend_name(&self) -> &'static str {
pub const fn driver_name(&self) -> &'static str {
match self {
Self::Docker { .. } => "docker",
Self::Podman { .. } => "podman",
Self::Kubernetes { .. } => "kubernetes-proxy-pod",
Self::Kubernetes { .. } => "kubernetes",
Self::Vm { .. } => "vm",
}
}
/// Validate the concrete fence properties and bind them to the selected
/// isolation backend.
pub fn validate_for_backend(&self, backend_name: &str) -> Result<(), BackendError> {
/// Validate the concrete outer-fence properties reported by the compute driver.
pub fn validate(&self) -> Result<(), BackendError> {
let valid = match self {
Self::Docker {
container_id,
network_mode,
unexpected_networks,
} => {
backend_name == "docker"
&& !container_id.is_empty()
&& network_mode == "none"
&& unexpected_networks.is_empty()
}
Self::Podman {
| Self::Podman {
container_id,
network_mode,
unexpected_networks,
} => {
backend_name == "podman"
&& !container_id.is_empty()
&& network_mode == "none"
&& unexpected_networks.is_empty()
!container_id.is_empty() && network_mode == "none" && unexpected_networks.is_empty()
}
Self::Kubernetes {
network_policy_uid,
@@ -499,8 +490,7 @@ impl DriverFenceEvidence {
egress_isolated,
egress_rule_count,
} => {
backend_name == "kubernetes-proxy-pod"
&& !network_policy_uid.is_empty()
!network_policy_uid.is_empty()
&& !network_policy_resource_version.is_empty()
&& *ingress_isolated
&& *egress_isolated
@@ -509,13 +499,14 @@ impl DriverFenceEvidence {
Self::Vm {
generation,
network_device_count,
} => backend_name == "vm" && !generation.is_empty() && *network_device_count == 0,
} => !generation.is_empty() && *network_device_count == 0,
};
if valid {
Ok(())
} else {
Err(BackendError::Confirm(format!(
"driver fence evidence is incomplete or does not match backend {backend_name:?}"
"{} driver fence evidence is incomplete",
self.driver_name()
)))
}
}
@@ -556,8 +547,7 @@ pub struct SandboxConfirmEvidence {
impl SandboxConfirmEvidence {
/// Validate the security-critical evidence required before launch.
pub fn validate(&self, expected: &ResolvedWorkloadIdentity) -> Result<(), BackendError> {
self.driver_fence
.validate_for_backend(self.driver_fence.backend_name())?;
self.driver_fence.validate()?;
let complete = &self.identity == expected
&& self.capabilities.is_empty()
&& self.no_new_privileges
@@ -297,7 +297,7 @@ impl<K: MockKind> IsolationBackend for MockBackend<K> {
}
async fn attach(
&self,
descriptor: VerifiedTopologyDescriptor,
descriptor: VerifiedBackendDescriptor,
sandbox: SandboxContext,
) -> Result<Box<dyn BoundBoundary>, BackendError> {
assert_eq!(descriptor.backend_name(), K::BACKEND_ID);
@@ -326,8 +326,8 @@ fn registry() -> BackendRegistry {
reg
}
fn descriptor(backend_name: &str) -> TopologyDescriptor {
TopologyDescriptor {
fn descriptor(backend_name: &str) -> BackendDescriptor {
BackendDescriptor {
backend_name: backend_name.to_string(),
payload: vec![],
}
@@ -432,28 +432,23 @@ fn driver_fence_evidence_is_backend_specific_and_fail_closed() {
network_device_count: 0,
};
assert!(docker.validate_for_backend("docker").is_ok());
assert!(
kubernetes
.validate_for_backend("kubernetes-proxy-pod")
.is_ok()
);
assert!(vm.validate_for_backend("vm").is_ok());
assert!(docker.validate_for_backend("vm").is_err());
assert!(docker.validate().is_ok());
assert!(kubernetes.validate().is_ok());
assert!(vm.validate().is_ok());
let drifted = DriverFenceEvidence::Docker {
container_id: "sha256:container".to_string(),
network_mode: "bridge".to_string(),
unexpected_networks: vec!["bridge".to_string()],
};
assert!(drifted.validate_for_backend("docker").is_err());
assert!(drifted.validate().is_err());
}
/// The backend-independent supervisor sequence. Identical for every backend:
/// this is the proof that adding a backend needs no supervisor lifecycle change.
async fn drive(
reg: &BackendRegistry,
descriptor: TopologyDescriptor,
descriptor: BackendDescriptor,
admitted: &str,
) -> Result<Box<dyn RunningBoundary>, BackendError> {
let (backend, verified) = reg.resolve(descriptor, admitted)?;
@@ -22,8 +22,8 @@ use openshell_core::policy::{
use openshell_isolation_interface::AgentSpec;
use openshell_isolation_interface::contract::Sha256Digest;
use openshell_isolation_interface::contract::{
BackendError, BinaryIdentity, BoundaryExitStatus, BoundarySignal, DriverFenceEvidence,
ExecSpec, ResolveError, SandboxConfirmEvidence, TopologyDescriptor,
BackendDescriptor, BackendError, BinaryIdentity, BoundaryExitStatus, BoundarySignal,
DriverFenceEvidence, ExecSpec, ResolveError, SandboxConfirmEvidence,
};
use rcgen::{CertificateParams, DnType, ExtendedKeyUsagePurpose, IsCa, KeyPair, KeyUsagePurpose};
use serde::de::DeserializeOwned;
@@ -185,10 +185,10 @@ pub struct GatewayVerificationKey {
pub public_key_pem: String,
}
/// Protected descriptor consumed by `openshell-supervisor`.
/// Protected runtime descriptor consumed by `openshell-supervisor`.
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct BoundaryTopology {
pub struct SandboxRuntimeDescriptor {
/// Stable identity of the boundary, normally the sandbox ID.
pub boundary_id: String,
/// Immutable driver-owned workload generation.
@@ -213,10 +213,10 @@ pub struct BoundaryTopology {
pub driver_fence: DriverFenceEvidence,
}
impl fmt::Debug for BoundaryTopology {
impl fmt::Debug for SandboxRuntimeDescriptor {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("BoundaryTopology")
.debug_struct("SandboxRuntimeDescriptor")
.field("boundary_id", &self.boundary_id)
.field("generation", &self.generation)
.field("session_id", &self.session_id)
@@ -229,17 +229,14 @@ impl fmt::Debug for BoundaryTopology {
}
}
impl BoundaryTopology {
/// Encode this topology as the shared RFC 0012 descriptor admitted for
/// `backend_name`.
pub fn descriptor(
&self,
backend_name: impl Into<String>,
) -> Result<TopologyDescriptor, BackendError> {
let payload = serde_json::to_vec(self)
.map_err(|error| BackendError::Descriptor(format!("encode topology: {error}")))?;
Ok(TopologyDescriptor {
backend_name: backend_name.into(),
impl SandboxRuntimeDescriptor {
/// Encode this runtime configuration for the `openshell-sandbox` backend.
pub fn backend_descriptor(&self) -> Result<BackendDescriptor, BackendError> {
let payload = serde_json::to_vec(self).map_err(|error| {
BackendError::Descriptor(format!("encode runtime descriptor: {error}"))
})?;
Ok(BackendDescriptor {
backend_name: crate::BACKEND_NAME.to_string(),
payload,
})
}
@@ -262,7 +259,7 @@ pub struct BoundaryConfig {
/// Driver-provisioned listener.
pub listener: BoundaryListener,
/// Immutable coordinates the boundary requires from the control-side
/// topology descriptor before accepting attachment.
/// runtime descriptor before accepting attachment.
#[serde(default)]
pub resource_claims: std::collections::BTreeMap<String, String>,
/// Driver-provisioned, read-only runtime evidence for resource claims.
@@ -15,6 +15,9 @@ pub mod sandbox_auth;
pub use runtime::OpenShellRuntimeBackend;
/// Stable isolation backend name implemented by `openshell-sandbox`.
pub const BACKEND_NAME: &str = "openshell-sandbox";
/// Generated gRPC transport envelope for the OpenShell Sandbox Protocol.
#[allow(
clippy::all,
+78 -75
View File
@@ -26,7 +26,7 @@ use openshell_isolation_interface::contract::{
BoundaryTerminal, ConfirmedBoundary, ExecSession, ExecSpec, IsolationBackend, LoopbackTarget,
MediationTiming, NetworkMediationSource, PendingDnsQuery, PendingTcpOpen, ProcessAttachment,
ReadyBoundary, RunningBoundary, SandboxContext, TcpOpenDecision, TcpOpenDenial,
VerifiedTopologyDescriptor,
VerifiedBackendDescriptor,
};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
#[cfg(unix)]
@@ -34,11 +34,11 @@ use tokio::net::UnixStream;
use tokio_stream::wrappers::ReceiverStream;
use crate::boundary_protocol::{
AgentSpecWire, BoundaryTopology, DnsQueryResultWire, ExecSpecWire, MAX_CONTROL_FRAME_BYTES,
Request, RequestEnvelope, Response, ResponseEnvelope, STREAM_EXIT, STREAM_STDERR, STREAM_STDIN,
STREAM_STDIN_CLOSED, STREAM_STDOUT, SandboxPolicyWire, SandboxTlsClientConfig,
SandboxTransport, SignalWire, decode_frame, encode_frame, read_stream_frame,
validate_resource_claims, write_stream_frame,
AgentSpecWire, DnsQueryResultWire, ExecSpecWire, MAX_CONTROL_FRAME_BYTES, Request,
RequestEnvelope, Response, ResponseEnvelope, STREAM_EXIT, STREAM_STDERR, STREAM_STDIN,
STREAM_STDIN_CLOSED, STREAM_STDOUT, SandboxPolicyWire, SandboxRuntimeDescriptor,
SandboxTlsClientConfig, SandboxTransport, SignalWire, decode_frame, encode_frame,
read_stream_frame, validate_resource_claims, write_stream_frame,
};
use crate::mediation::{self, DnsQueryWire, MediationFrame, MediationFrameKind};
@@ -51,7 +51,6 @@ const CONNECT_RETRY_TIMEOUT: Duration = Duration::from_secs(30);
/// Host-side `OpenShell` Sandbox Protocol implementation registered with the supervisor.
#[derive(Debug)]
pub struct OpenShellRuntimeBackend {
backend_name: String,
ca_file_paths: Arc<std::sync::Mutex<Option<(PathBuf, PathBuf)>>>,
provider_credentials: openshell_core::provider_credentials::ProviderCredentialState,
sandbox_bearer: openshell_core::jwt::SessionBearerTokenSlot,
@@ -59,13 +58,11 @@ pub struct OpenShellRuntimeBackend {
impl OpenShellRuntimeBackend {
pub fn new(
backend_name: impl Into<String>,
ca_file_paths: Arc<std::sync::Mutex<Option<(PathBuf, PathBuf)>>>,
provider_credentials: openshell_core::provider_credentials::ProviderCredentialState,
sandbox_bearer: openshell_core::jwt::SessionBearerTokenSlot,
) -> Self {
Self {
backend_name: backend_name.into(),
ca_file_paths,
provider_credentials,
sandbox_bearer,
@@ -76,23 +73,28 @@ impl OpenShellRuntimeBackend {
#[async_trait]
impl IsolationBackend for OpenShellRuntimeBackend {
fn backend_name(&self) -> &str {
&self.backend_name
crate::BACKEND_NAME
}
async fn attach(
&self,
descriptor: VerifiedTopologyDescriptor,
descriptor: VerifiedBackendDescriptor,
sandbox: SandboxContext,
) -> Result<Box<dyn BoundBoundary>, BackendError> {
let topology: BoundaryTopology = serde_json::from_slice(descriptor.payload())
.map_err(|error| BackendError::Descriptor(format!("decode topology: {error}")))?;
validate_topology(&topology, &sandbox, &self.backend_name)?;
let host_gateway_ip = topology.host_gateway_ip;
let resource_claims = topology.resource_claims.clone();
let generation = topology.generation.clone();
let session_id = topology.session_id;
let driver_fence = topology.driver_fence.clone();
let client = Arc::new(BoundaryClient::new(topology, self.sandbox_bearer.clone()));
let runtime_descriptor: SandboxRuntimeDescriptor =
serde_json::from_slice(descriptor.payload()).map_err(|error| {
BackendError::Descriptor(format!("decode runtime descriptor: {error}"))
})?;
validate_runtime_descriptor(&runtime_descriptor, &sandbox)?;
let host_gateway_ip = runtime_descriptor.host_gateway_ip;
let resource_claims = runtime_descriptor.resource_claims.clone();
let generation = runtime_descriptor.generation.clone();
let session_id = runtime_descriptor.session_id;
let driver_fence = runtime_descriptor.driver_fence.clone();
let client = Arc::new(BoundaryClient::new(
runtime_descriptor,
self.sandbox_bearer.clone(),
));
let response = client
.call_idempotent(Request::Attach {
policy: Box::new(SandboxPolicyWire::from(sandbox.policy.clone())),
@@ -104,7 +106,7 @@ impl IsolationBackend for OpenShellRuntimeBackend {
};
if snapshot.generation != generation {
return Err(BackendError::Confirm(
"sandbox session snapshot generation does not match topology".to_string(),
"sandbox session snapshot generation does not match runtime descriptor".to_string(),
));
}
Ok(Box::new(RemoteBound {
@@ -125,35 +127,35 @@ impl IsolationBackend for OpenShellRuntimeBackend {
}
}
fn validate_topology(
topology: &BoundaryTopology,
fn validate_runtime_descriptor(
runtime_descriptor: &SandboxRuntimeDescriptor,
sandbox: &SandboxContext,
backend_name: &str,
) -> Result<(), BackendError> {
if topology.boundary_id != sandbox.sandbox_id {
if runtime_descriptor.boundary_id != sandbox.sandbox_id {
return Err(BackendError::Descriptor(format!(
"boundary {:?} does not match sandbox {:?}",
topology.boundary_id, sandbox.sandbox_id
runtime_descriptor.boundary_id, sandbox.sandbox_id
)));
}
if topology.generation.is_empty() {
if runtime_descriptor.generation.is_empty() {
return Err(BackendError::Descriptor(
"boundary generation must not be empty".to_string(),
));
}
if topology.session_id != sandbox.session_id {
if runtime_descriptor.session_id != sandbox.session_id {
return Err(BackendError::Descriptor(
"topology session ID does not match admitted sandbox session".to_string(),
"runtime descriptor session ID does not match admitted sandbox session".to_string(),
));
}
if topology.workload_identity != sandbox.identity {
if runtime_descriptor.workload_identity != sandbox.identity {
return Err(BackendError::Descriptor(
"topology workload identity does not match admitted sandbox identity".to_string(),
"runtime descriptor workload identity does not match admitted sandbox identity"
.to_string(),
));
}
validate_resource_claims(&topology.resource_claims)?;
topology.driver_fence.validate_for_backend(backend_name)?;
match &topology.transport {
validate_resource_claims(&runtime_descriptor.resource_claims)?;
runtime_descriptor.driver_fence.validate()?;
match &runtime_descriptor.transport {
SandboxTransport::Unix { socket_path } => {
validate_socket_path(socket_path)?;
}
@@ -180,7 +182,7 @@ fn validate_topology(
validate_control_port(*port)?;
}
}
validate_client_tls(&topology.tls)?;
validate_client_tls(&runtime_descriptor.tls)?;
Ok(())
}
@@ -287,7 +289,7 @@ impl BoundBoundary for RemoteBound {
|| evidence.driver_fence != self.driver_fence
{
return Err(BackendError::Confirm(
"sandbox confirmation generation, session, resource claims, or driver fence do not match topology"
"sandbox confirmation generation, session, resource claims, or driver fence do not match runtime descriptor"
.to_string(),
));
}
@@ -945,7 +947,7 @@ async fn dispatch_client_mediation_frame(
}
struct BoundaryClient {
topology: BoundaryTopology,
runtime_descriptor: SandboxRuntimeDescriptor,
sandbox_bearer: openshell_core::jwt::SessionBearerTokenSlot,
grpc_channel: tokio::sync::Mutex<Option<CachedGrpcChannel>>,
mediation: tokio::sync::Mutex<Option<Arc<ClientMediationSession>>>,
@@ -965,11 +967,11 @@ struct CachedGrpcChannel {
impl BoundaryClient {
fn new(
topology: BoundaryTopology,
runtime_descriptor: SandboxRuntimeDescriptor,
sandbox_bearer: openshell_core::jwt::SessionBearerTokenSlot,
) -> Self {
Self {
topology,
runtime_descriptor,
sandbox_bearer,
grpc_channel: tokio::sync::Mutex::new(None),
mediation: tokio::sync::Mutex::new(None),
@@ -1419,7 +1421,7 @@ impl BoundaryClient {
}
async fn build_grpc_channel(&self) -> Result<tonic::transport::Channel, BackendError> {
let topology = self.topology.clone();
let runtime_descriptor = self.runtime_descriptor.clone();
let endpoint =
tonic::transport::Endpoint::from_static("http://boundary.openshell.internal")
.initial_stream_window_size(16 * 1024 * 1024)
@@ -1428,9 +1430,9 @@ impl BoundaryClient {
.keep_alive_while_idle(true);
let channel = endpoint
.connect_with_connector(tower::service_fn(move |_: tonic::transport::Uri| {
let topology = topology.clone();
let runtime_descriptor = runtime_descriptor.clone();
async move {
connect_boundary_with_retry(&topology)
connect_boundary_with_retry(&runtime_descriptor)
.await
.map(TokioIo::new)
.map_err(|error| std::io::Error::other(error.to_string()))
@@ -1445,16 +1447,16 @@ impl BoundaryClient {
#[cfg(test)]
async fn connect_boundary_once(&self) -> Result<BoundaryDuplexStream, BackendError> {
connect_boundary_once(&self.topology).await
connect_boundary_once(&self.runtime_descriptor).await
}
}
async fn connect_boundary_with_retry(
topology: &BoundaryTopology,
runtime_descriptor: &SandboxRuntimeDescriptor,
) -> Result<BoundaryDuplexStream, BackendError> {
let deadline = tokio::time::Instant::now() + CONNECT_RETRY_TIMEOUT;
loop {
match connect_boundary_once(topology).await {
match connect_boundary_once(runtime_descriptor).await {
Ok(stream) => return Ok(stream),
Err(error) if tokio::time::Instant::now() >= deadline => return Err(error),
Err(_) => tokio::time::sleep(Duration::from_millis(25)).await,
@@ -1463,9 +1465,9 @@ async fn connect_boundary_with_retry(
}
async fn connect_boundary_once(
topology: &BoundaryTopology,
runtime_descriptor: &SandboxRuntimeDescriptor,
) -> Result<BoundaryDuplexStream, BackendError> {
let stream: BoundaryDuplexStream = match &topology.transport {
let stream: BoundaryDuplexStream = match &runtime_descriptor.transport {
#[cfg(unix)]
SandboxTransport::Unix { socket_path } => {
let stream = UnixStream::connect(socket_path).await.map_err(|error| {
@@ -1498,7 +1500,7 @@ async fn connect_boundary_once(
}
SandboxTransport::Vsock { guest_cid, port } => connect_host_vsock(*guest_cid, *port)?,
};
let tls = &topology.tls;
let tls = &runtime_descriptor.tls;
let server_name =
rustls::pki_types::ServerName::try_from(tls.server_name.clone()).map_err(|error| {
BackendError::Descriptor(format!(
@@ -1918,7 +1920,7 @@ mod tests {
.await
.unwrap();
let client = BoundaryClient::new(
tls_topology(address, test_certificate().client_tls),
tls_runtime_descriptor(address, test_certificate().client_tls),
test_bearer(&"a".repeat(32)),
);
*client.grpc_channel.lock().await = Some(CachedGrpcChannel {
@@ -1974,7 +1976,7 @@ mod tests {
}
});
let client = BoundaryClient::new(
tls_topology(address, certificate.client_tls),
tls_runtime_descriptor(address, certificate.client_tls),
test_bearer(&"a".repeat(32)),
);
let attach = Request::Attach {
@@ -2119,11 +2121,11 @@ mod tests {
}
}
fn tls_topology(
fn tls_runtime_descriptor(
address: std::net::SocketAddr,
tls: SandboxTlsClientConfig,
) -> BoundaryTopology {
BoundaryTopology {
) -> SandboxRuntimeDescriptor {
SandboxRuntimeDescriptor {
boundary_id: "sandbox-1".to_string(),
generation: "test-generation".to_string(),
session_id: test_session_id(),
@@ -2261,9 +2263,9 @@ mod tests {
}
#[test]
fn topology_debug_redacts_trust_anchor() {
fn runtime_descriptor_debug_redacts_trust_anchor() {
let certificate = test_certificate();
let topology = BoundaryTopology {
let runtime_descriptor = SandboxRuntimeDescriptor {
boundary_id: "sandbox-1".to_string(),
generation: "test-generation".to_string(),
session_id: test_session_id(),
@@ -2276,14 +2278,14 @@ mod tests {
resource_claims: std::collections::BTreeMap::new(),
driver_fence: test_driver_fence(),
};
let debug = format!("{topology:?}");
let debug = format!("{runtime_descriptor:?}");
assert!(debug.contains("<redacted>"));
assert!(!debug.contains(&certificate.client_tls.trust_anchor_pem));
}
#[test]
fn topology_must_match_sandbox() {
let topology = BoundaryTopology {
fn runtime_descriptor_must_match_sandbox() {
let runtime_descriptor = SandboxRuntimeDescriptor {
boundary_id: "other".to_string(),
generation: "test-generation".to_string(),
session_id: test_session_id(),
@@ -2297,14 +2299,14 @@ mod tests {
driver_fence: test_driver_fence(),
};
assert!(matches!(
validate_topology(&topology, &sandbox(), "vm"),
validate_runtime_descriptor(&runtime_descriptor, &sandbox()),
Err(BackendError::Descriptor(_))
));
}
#[test]
fn topology_rejects_an_unspecified_tcp_target() {
let topology = BoundaryTopology {
fn runtime_descriptor_rejects_an_unspecified_tcp_target() {
let runtime_descriptor = SandboxRuntimeDescriptor {
boundary_id: "sandbox-1".to_string(),
generation: "test-generation".to_string(),
session_id: test_session_id(),
@@ -2319,14 +2321,14 @@ mod tests {
driver_fence: test_driver_fence(),
};
assert!(matches!(
validate_topology(&topology, &sandbox(), "vm"),
validate_runtime_descriptor(&runtime_descriptor, &sandbox()),
Err(BackendError::Descriptor(_))
));
}
#[test]
fn topology_accepts_a_concrete_tcp_target() {
let topology = BoundaryTopology {
fn runtime_descriptor_accepts_a_concrete_tcp_target() {
let runtime_descriptor = SandboxRuntimeDescriptor {
boundary_id: "sandbox-1".to_string(),
generation: "test-generation".to_string(),
session_id: test_session_id(),
@@ -2340,12 +2342,13 @@ mod tests {
resource_claims: std::collections::BTreeMap::new(),
driver_fence: test_driver_fence(),
};
validate_topology(&topology, &sandbox(), "vm").expect("TCP topology should be valid");
validate_runtime_descriptor(&runtime_descriptor, &sandbox())
.expect("TCP runtime descriptor should be valid");
}
#[test]
fn topology_rejects_invalid_tls_configuration() {
let topology = tls_topology(
fn runtime_descriptor_rejects_invalid_tls_configuration() {
let runtime_descriptor = tls_runtime_descriptor(
"127.0.0.1:5500".parse().expect("valid address"),
SandboxTlsClientConfig {
server_name: "not a dns name!".to_string(),
@@ -2353,7 +2356,7 @@ mod tests {
},
);
assert!(matches!(
validate_topology(&topology, &sandbox(), "vm"),
validate_runtime_descriptor(&runtime_descriptor, &sandbox()),
Err(BackendError::Descriptor(_))
));
}
@@ -2363,7 +2366,7 @@ mod tests {
let certificate = test_certificate();
let (address, server) = spawn_tls_boundary(certificate.server_config, "a".repeat(32)).await;
let client = BoundaryClient::new(
tls_topology(address, certificate.client_tls),
tls_runtime_descriptor(address, certificate.client_tls),
test_bearer(&"a".repeat(32)),
);
@@ -2405,7 +2408,7 @@ mod tests {
let certificate = test_certificate();
let (address, server) = spawn_tls_boundary(certificate.server_config, "a".repeat(32)).await;
let client = Arc::new(BoundaryClient::new(
tls_topology(address, certificate.client_tls),
tls_runtime_descriptor(address, certificate.client_tls),
test_bearer(&"a".repeat(32)),
));
let session = open_exec_session(
@@ -2516,7 +2519,7 @@ mod tests {
}
});
let client = Arc::new(BoundaryClient::new(
BoundaryTopology {
SandboxRuntimeDescriptor {
boundary_id: "sandbox-1".to_string(),
generation: "test-generation".to_string(),
session_id: test_session_id(),
@@ -2557,7 +2560,7 @@ mod tests {
let certificate = test_certificate();
let (address, server) = spawn_tls_boundary(certificate.server_config, "a".repeat(32)).await;
let client = BoundaryClient::new(
tls_topology(address, certificate.client_tls),
tls_runtime_descriptor(address, certificate.client_tls),
test_bearer(&"a".repeat(32)),
);
let context = sandbox();
@@ -2604,7 +2607,7 @@ mod tests {
});
let context = sandbox();
let client = BoundaryClient::new(
BoundaryTopology {
SandboxRuntimeDescriptor {
boundary_id: "sandbox-1".to_string(),
generation: "test-generation".to_string(),
session_id: test_session_id(),
@@ -2651,7 +2654,7 @@ mod tests {
)
.await;
let client = BoundaryClient::new(
tls_topology(address, certificate.client_tls),
tls_runtime_descriptor(address, certificate.client_tls),
test_bearer("incorrect-token-incorrect-token"),
);
@@ -2668,7 +2671,7 @@ mod tests {
let trusted = test_certificate();
let (address, server) = spawn_tls_boundary(presented.server_config, "a".repeat(32)).await;
let client = BoundaryClient::new(
tls_topology(address, trusted.client_tls),
tls_runtime_descriptor(address, trusted.client_tls),
test_bearer(&"a".repeat(32)),
);
@@ -221,7 +221,7 @@ mod linux {
validate_resource_claims(&config.resource_claims).map_err(|error| error.to_string())?;
config
.driver_fence
.validate_for_backend(config.driver_fence.backend_name())
.validate()
.map_err(|error| error.to_string())?;
for (claim, path) in &config.resource_claim_files {
if !config.resource_claims.contains_key(claim) {
@@ -1653,7 +1653,7 @@ mod linux {
} else {
guest_error(
BoundaryErrorKind::Denied,
"topology resource claims do not match the boundary configuration",
"backend resource claims do not match the boundary configuration",
)
}
}
+11 -12
View File
@@ -309,7 +309,7 @@ pub async fn run_sandbox(
health_socket_path: Option<std::path::PathBuf>,
ocsf_enabled: Arc<AtomicBool>,
upstream_proxy_args: openshell_supervisor_network::upstream_proxy::UpstreamProxyArgs,
topology_descriptor: openshell_isolation_interface::contract::TopologyDescriptor,
backend_descriptor: openshell_isolation_interface::contract::BackendDescriptor,
auth_bundle: openshell_core::jwt::SupervisorAuthBundle,
admitted_isolation_backend: Option<String>,
main_exit_marker: Option<std::path::PathBuf>,
@@ -497,26 +497,25 @@ pub async fn run_sandbox(
// the entrypoint process's /proc/net/tcp for identity binding.
let entrypoint_pid = Arc::new(AtomicU32::new(0));
// A separated topology uses the shared authenticated boundary protocol.
// The sandbox runtime uses the shared authenticated boundary protocol.
// The admitted backend name is resolved independently of the protected
// descriptor, and generic supervisor code never imports a driver crate.
let admitted_backend_name = admitted_isolation_backend.ok_or_else(|| {
miette::miette!("protected topology supplied without an admitted isolation backend")
miette::miette!("runtime descriptor supplied without an admitted isolation backend")
})?;
let topology: openshell_sandbox_backend::boundary_protocol::BoundaryTopology =
serde_json::from_slice(&topology_descriptor.payload)
.map_err(|error| miette::miette!("decode boundary topology: {error}"))?;
if auth_bundle.session_id != topology.session_id {
let runtime_descriptor: openshell_sandbox_backend::boundary_protocol::SandboxRuntimeDescriptor =
serde_json::from_slice(&backend_descriptor.payload)
.map_err(|error| miette::miette!("decode sandbox runtime descriptor: {error}"))?;
if auth_bundle.session_id != runtime_descriptor.session_id {
return Err(miette::miette!(
"supervisor authentication bundle does not match topology session"
"supervisor authentication bundle does not match runtime session"
));
}
let sandbox_bearer = openshell_core::grpc_client::install_supervisor_auth_bundle(&auth_bundle)?;
let session_id = topology.session_id;
let session_id = runtime_descriptor.session_id;
let ca_file_paths = Arc::new(std::sync::Mutex::new(None));
let backend: Arc<dyn openshell_isolation_interface::contract::IsolationBackend> =
Arc::new(openshell_sandbox_backend::OpenShellRuntimeBackend::new(
admitted_backend_name.clone(),
ca_file_paths.clone(),
provider_credentials.clone(),
sandbox_bearer,
@@ -526,7 +525,7 @@ pub async fn run_sandbox(
.register(backend)
.map_err(|error| miette::miette!(error.to_string()))?;
let (backend, verified) = registry
.resolve(topology_descriptor, &admitted_backend_name)
.resolve(backend_descriptor, &admitted_backend_name)
.map_err(|error| miette::miette!(error.to_string()))?;
let context = openshell_isolation_interface::contract::SandboxContext {
sandbox_id: sandbox_id.clone().unwrap_or_default(),
@@ -539,7 +538,7 @@ pub async fn run_sandbox(
timeout_secs,
interactive,
},
identity: topology.workload_identity,
identity: runtime_descriptor.workload_identity,
};
let bound = backend
.attach(verified, context)
+25 -26
View File
@@ -9,7 +9,7 @@ use std::sync::atomic::AtomicBool;
use clap::Parser;
use miette::{IntoDiagnostic, Result};
use openshell_isolation_interface::contract::TopologyDescriptor;
use openshell_isolation_interface::contract::BackendDescriptor;
use openshell_ocsf::{OcsfJsonlLayer, OcsfShorthandLayer};
use tracing::{info, warn};
use tracing_subscriber::EnvFilter;
@@ -94,10 +94,7 @@ struct Args {
upstream_proxy_ca_bundle: Option<String>,
#[arg(long)]
topology_backend_name: String,
#[arg(long)]
topology_payload_file: PathBuf,
backend_descriptor_file: PathBuf,
/// Protected gateway-issued credentials for this exact sandbox launch.
#[arg(long)]
@@ -107,15 +104,15 @@ struct Args {
main_exit_marker: Option<PathBuf>,
}
fn topology(args: &Args) -> Result<TopologyDescriptor> {
let payload = std::fs::read(&args.topology_payload_file).map_err(|error| {
fn backend_descriptor(args: &Args) -> Result<BackendDescriptor> {
let payload = std::fs::read(&args.backend_descriptor_file).map_err(|error| {
miette::miette!(
"read topology payload {}: {error}",
args.topology_payload_file.display()
"read backend descriptor {}: {error}",
args.backend_descriptor_file.display()
)
})?;
Ok(TopologyDescriptor {
backend_name: args.topology_backend_name.clone(),
Ok(BackendDescriptor {
backend_name: openshell_sandbox_backend::BACKEND_NAME.to_string(),
payload,
})
}
@@ -166,7 +163,7 @@ fn main() -> Result<()> {
let args = Args::parse();
validate_main_exit_marker(args.main_exit_marker.as_deref())?;
let topology = topology(&args)?;
let backend_descriptor = backend_descriptor(&args)?;
let auth_bundle = auth_bundle(&args)?;
let file_logging = tracing_appender::rolling::RollingFileAppender::builder()
@@ -297,7 +294,7 @@ fn main() -> Result<()> {
args.health_socket_path,
ocsf_enabled,
upstream_proxy_args,
topology,
backend_descriptor,
auth_bundle,
admitted_isolation_backend,
args.main_exit_marker,
@@ -314,30 +311,32 @@ mod tests {
#[test]
fn role_specific_cli_has_no_mode_switch() {
let directory = tempfile::tempdir().expect("temporary topology directory");
let topology_path = directory.path().join("topology.json");
let directory = tempfile::tempdir().expect("temporary runtime descriptor directory");
let descriptor_path = directory.path().join("runtime-descriptor.json");
let auth_bundle_path = directory.path().join("auth-bundle.json");
std::fs::write(&topology_path, [0]).expect("write topology payload");
std::fs::write(&descriptor_path, [0]).expect("write runtime descriptor payload");
std::fs::write(&auth_bundle_path, [0]).expect("write auth bundle payload");
let args = Args::try_parse_from([
"openshell-supervisor",
"--topology-backend-name",
"test",
"--topology-payload-file",
topology_path.to_str().expect("UTF-8 topology path"),
"--backend-descriptor-file",
descriptor_path
.to_str()
.expect("UTF-8 runtime descriptor path"),
"--auth-bundle-file",
auth_bundle_path.to_str().expect("UTF-8 auth bundle path"),
])
.expect("supervisor arguments");
assert_eq!(topology(&args).expect("topology").payload, vec![0]);
assert_eq!(
backend_descriptor(&args)
.expect("runtime descriptor")
.payload,
vec![0]
);
}
#[test]
fn topology_payload_is_mandatory() {
assert!(
Args::try_parse_from(["openshell-supervisor", "--topology-backend-name", "test"])
.is_err()
);
fn backend_descriptor_is_mandatory() {
assert!(Args::try_parse_from(["openshell-supervisor"]).is_err());
}
#[test]
+48 -48
View File
@@ -19,7 +19,7 @@ links:
Today the supervisor both builds the workload's isolation boundary and applies its network policy. Because the supervisor runs inside the agent container, the privilege needed to build that boundary sits beside the code it confines. This RFC moves boundary construction and process operations behind a pluggable **Isolation Backend**. The supervisor continues to apply approved network policy through network mediation.
The compute driver provisions the workload and trusted components. The logical supervisor is the trusted bridge between the gateway and the workload: it maintains the gateway connection, handles authorized requests, and drives the backend. The backend establishes the isolation controls, manages workload processes, and routes egress to network mediation. The same lifecycle supports today's in-pod implementation and future delegated implementations without topology-specific supervisor paths.
The compute driver orchestrates the workload, supervisor, sandbox runtime, protected channel, and outer network fence. The supervisor is the trusted bridge between the gateway and the workload: it maintains the gateway connection, handles authorized requests, and drives the backend. The backend establishes the isolation controls, manages workload processes, and routes egress to network mediation. The same lifecycle supports different compute drivers and backend implementations without driver-specific supervisor paths.
## Motivation
@@ -33,7 +33,7 @@ All three come from coupling boundary construction to boundary operation. A comm
## Non-goals
- **Implementing a delegated backend.** Each topology requires its own design and implementation.
- **Implementing a delegated backend.** Each backend implementation and compute-driver placement requires its own design and implementation.
- **Changing authorization.** [RFC 0001](../0001-core-architecture/README.md) owns control-plane and sandbox identity. A delegated backend must still authenticate callers and scope them to one boundary.
- **Standardizing backend-internal component coordination.** A backend may coordinate helper, sidecar, or interception processes behind one lifecycle; how those components cooperate is backend-specific, not contract surface.
- **Changing gateway lifecycle or public status.** This RFC adds no gateway activation operation, public phase, or status API, and it does not define how a boundary's effective isolation model is surfaced to operators.
@@ -42,23 +42,23 @@ All three come from coupling boundary construction to boundary operation. A comm
The mental model has three roles:
- The **compute driver** provisions the sandbox instance according to the selected placement of the workload and trusted isolation components. That placement is the **topology**.
- The **Isolation Backend** establishes and operates the topology-specific controls around the workload. It also routes workload egress to network mediation and provides process operations.
- The **compute driver** orchestrates the sandbox instance: it places the workload, supervisor, and sandbox runtime; creates their protected channel; and applies the outer network fence.
- The **Isolation Backend** establishes and operates the controls around the workload. It also routes workload egress to network mediation and provides process operations.
- The **logical supervisor** is the trusted control-plane bridge between the gateway and the workload. It drives the backend, handles authorized gateway requests, and applies approved network policy through network mediation.
Together, network policy, filesystem isolation, syscall filtering, and sandbox identity form the workload's isolation boundary. The roles above enforce that boundary and may run in one process or across several trusted components. Their placement does not change the contract.
Each active boundary has at most one logical supervisor, which may span multiple coupled processes. The backend routes all workload egress through a per-boundary source, and the supervisor consumes that source. Internal delegation and transport remain topology-private.
Each active boundary has at most one logical supervisor, which may span multiple coupled processes. The backend routes all workload egress through a per-boundary source, and the supervisor consumes that source. Internal delegation and transport remain backend-private.
[RFC 0001](../0001-core-architecture/README.md) continues to own sandbox authentication and authorization. In this contract, sandbox identity means binding the authenticated sandbox context to the isolation boundary.
Admission selects the sandbox's topology and determines its trusted context. The compute driver sets up the topology and gives the logical supervisor a `TopologyDescriptor` describing what it provisioned. The supervisor uses the descriptor to attach the matching Isolation Backend. The backend prepares the required controls before the agent starts.
Admission selects the sandbox's Isolation Backend and determines its trusted context. The compute driver orchestrates the required resources and gives the supervisor a `BackendDescriptor` describing the selected backend and provisioned runtime. The supervisor uses the descriptor to attach the matching Isolation Backend. The backend prepares the required controls before the agent starts.
```mermaid
flowchart TB
Gateway["Gateway"] -->|"create sandbox"| Driver["Compute driver"]
subgraph Topology["Driver-provisioned topology (placement varies)"]
subgraph Runtime["Driver-orchestrated sandbox"]
Supervisor["Supervisor"]
Backend["Isolation Backend (may coordinate components)"]
subgraph Boundary["Isolation boundary"]
@@ -76,13 +76,13 @@ flowchart TB
Workload ==>|"only egress"| Mediator
end
Driver -->|"resources + TopologyDescriptor"| Supervisor
Driver -->|"resources + BackendDescriptor"| Supervisor
Mediator -->|"allowed egress"| Egress["Egress"]
```
In the in-pod topology, the supervisor drives a backend implemented in the same process. Other topologies may delegate backend operations without changing the supervisor lifecycle.
The initial implementation is `OpenShellRuntimeBackend`: a supervisor-side client that delegates operations over the authenticated OpenShell Sandbox Protocol to `openshell-sandbox` beside the workload. A future runtime-native backend could implement the same interface without using that protocol or binary.
A boundary is active from successful `attach` until normal backend cleanup releases the binding or the topology's trusted cleanup path invalidates it. A backend may coordinate multiple trusted helper or interception processes for that boundary. The backend owns the active-boundary binding; the compute driver owns the sandbox instance and topology lifecycle.
A boundary is active from successful `attach` until normal backend cleanup releases the binding or the compute driver's trusted cleanup path invalidates it. A backend may coordinate multiple trusted helper or interception processes for that boundary. The backend owns the active-boundary binding; the compute driver owns the sandbox instance and infrastructure lifecycle.
### Contract invariants
@@ -99,52 +99,52 @@ Each backend states its termination bound in its implementation documentation. L
### Provisioning
Provisioning runs on the control plane, and three rules hold in every topology:
Provisioning runs on the control plane, and three rules hold for every sandbox:
1. **Admission selects the topology** from trusted deployment configuration, not `SandboxPolicy`, and records its required backend. The `TopologyDescriptor` supplied by the compute driver must name that backend, and resolution never falls back to another backend.
2. **The compute driver provisions the topology** and anything the selected backend needs.
1. **Admission selects the Isolation Backend** from trusted deployment configuration, not `SandboxPolicy`. The `BackendDescriptor` supplied by the compute driver must name that backend, and resolution never falls back to another backend.
2. **The compute driver orchestrates the sandbox** and anything the selected backend needs.
3. **The backend establishes standing enforcement before untrusted code runs**, during provisioning or `attach`, depending on the backend.
If a topology depends on cluster-scoped coverage or registration, admission verifies that the prerequisite covers the boundary's placement before untrusted code runs.
If a driver depends on cluster-scoped coverage or registration, admission verifies that the prerequisite covers the boundary's placement before untrusted code runs.
Every topology provides a trusted cleanup path that does not depend on logical-supervisor availability.
Every compute driver provides a trusted cleanup path that does not depend on supervisor availability.
A compute driver may provision a resource and `TopologyDescriptor` before the control plane assigns it to a sandbox. No untrusted workload runs while the resource is unassigned. After claim or assignment produces a trusted `SandboxContext`, the supervisor calls `attach`; the backend either binds that context to the prepared resource and returns `Bound`, or rejects it as incompatible. Pool creation, claim, reset, release, and recycling remain outside this contract.
A compute driver may provision a resource and `BackendDescriptor` before the control plane assigns it to a sandbox. No untrusted workload runs while the resource is unassigned. After claim or assignment produces a trusted `SandboxContext`, the supervisor calls `attach`; the backend either binds that context to the prepared resource and returns `Bound`, or rejects it as incompatible. Pool creation, claim, reset, release, and recycling remain outside this contract.
Every create or start-from-stopped operation receives a fresh `SandboxSessionId`. Retries of the same durable launch transition reuse that identity, while a later launch receives a new one even when the compute platform reuses its outer resource. The backend binds `attach` to that exact session and rejects descriptors or runtime sessions left over from an earlier launch.
### The topology descriptor
### The backend descriptor
The driver supplies a descriptor for every provisioned topology, including in-pod and resources prepared before assignment. The common envelope names the backend and carries an opaque payload.
The driver supplies a descriptor for every provisioned sandbox, including resources prepared before assignment. The common envelope names the backend and carries an opaque payload.
```rust
struct TopologyDescriptor {
struct BackendDescriptor {
backend_name: String,
version: u32,
payload: Vec<u8>,
}
```
`version` is the Isolation Backend interface version. Backend name and version match exactly; this contract does not negotiate compatibility ranges. The descriptor is transport-neutral. Provisioning supplies it to the supervisor before `attach`; how it is transported is topology-specific and outside this contract, and every transport preserves one property: workload-controlled input cannot select or modify the descriptor.
`version` is the Isolation Backend interface version. Backend name and version match exactly; this contract does not negotiate compatibility ranges. The descriptor is transport-neutral. Provisioning supplies it to the supervisor before `attach`; how it is transported is compute-driver-specific and outside this contract, and every transport preserves one property: workload-controlled input cannot select or modify the descriptor.
The opaque payload identifies, or gives the backend enough information to resolve, the exact driver-provisioned resource. It may also carry topology-specific endpoint or helper-role information; there are no common topology or role fields.
The opaque payload identifies, or gives the backend enough information to resolve, the exact driver-provisioned resource. It may also carry backend-specific endpoint or helper-role information; there are no common placement or role fields.
Common verification requires:
- the descriptor's `backend_name` matches the backend required by the admitted topology;
- the descriptor's `backend_name` matches the backend selected during admission;
- the descriptor's version is one the supervisor supports, and the resolved backend reports that same version; and
- `SandboxContext` is constructed after the control plane assigns the resource to the admitted sandbox, using authenticated control-plane and trusted supervisor state.
The supervisor validates the descriptor's common fields and produces a `VerifiedTopologyDescriptor`, then resolves its `backend_name` and version without fallback. Verification does not imply that the opaque payload is valid; the selected backend validates it and atomically binds the provisioned resource to the trusted `SandboxContext` during `attach`. Any failure rejects the sandbox.
The supervisor validates the descriptor's common fields and produces a `VerifiedBackendDescriptor`, then resolves its `backend_name` and version without fallback. Verification does not imply that the opaque payload is valid; the selected backend validates it and atomically binds the provisioned resource to the trusted `SandboxContext` during `attach`. Any failure rejects the sandbox.
### The lifecycle
The contract does not prescribe enforcement mechanisms; it standardizes how the supervisor drives whichever backend a deployment admits.
A backend registers under a `backend_name` and version. The supervisor attaches to the admitted topology and drives the boundary through a fixed sequence of states. Each transition consumes the prior state, so the supervisor cannot skip a stage or invoke a later transition through an earlier handle. The Rust names are illustrative; the states and their semantics are normative.
A backend registers under a `backend_name` and version. The supervisor attaches to the admitted sandbox and drives the boundary through a fixed sequence of states. Each transition consumes the prior state, so the supervisor cannot skip a stage or invoke a later transition through an earlier handle. The Rust names are illustrative; the states and their semantics are normative.
```text
attach topology + sandbox context -> Bound -> confirm -> Ready -> start_agent -> Running
attach backend descriptor + sandbox context -> Bound -> confirm -> Ready -> start_agent -> Running
```
```rust
@@ -155,7 +155,7 @@ trait IsolationBackend: Send + Sync {
async fn attach(
&self,
descriptor: VerifiedTopologyDescriptor,
descriptor: VerifiedBackendDescriptor,
sandbox: SandboxContext,
) -> Result<Box<dyn BoundBoundary>, BackendError>;
}
@@ -197,7 +197,7 @@ trait RunningBoundary: Send + Sync {
The states have normative meanings:
- **Bound:** the topology descriptor and trusted sandbox context are bound to the same resource, and the network-mediation source is available. No untrusted workload code is running.
- **Bound:** the backend descriptor and trusted sandbox context are bound to the same resource, and the network-mediation source is available. No untrusted workload code is running.
- **Ready:** the backend has confirmed standing enforcement for this concrete boundary and is prepared to apply the admitted launch-time controls before untrusted execution.
- **Running:** `start_agent` has made the admitted agent runnable and returned `RunningBoundary`. Every applicable launch-time control was in force before the first untrusted instruction. Whether the backend creates the agent process or releases a held, driver-provisioned execution object is backend-specific; the contract fixes the ordering, not the mechanism.
@@ -277,7 +277,7 @@ struct PendingDnsQuery {
`NetworkMediationSource` supplies typed TCP opens and DNS queries from one boundary to supervisor-owned network mediation. The two accepts are independent so the supervisor can consume them concurrently without coupling their ordering or backpressure. The backend routes all workload egress through these sources and authoritatively associates each request with the boundary without relying solely on workload-provided data. Capture, transport, framing, placement, and coordination are backend-private. General UDP mediation is outside this contract.
Every topology may use the same supervisor-owned mediation libraries or services; the source does not require a backend-specific policy engine.
Every backend may use the same supervisor-owned mediation libraries or services; the source does not require a backend-specific policy engine.
Shared implementations isolate each boundary's state and enforcement. Failure or teardown of one boundary cannot weaken another. Network-mediation unavailability never enables direct egress.
@@ -308,14 +308,14 @@ The logical supervisor resolves `backend_name` and version through a trusted imp
The supervisor runs the same sequence for every backend:
1. Obtain the `TopologyDescriptor` and trusted `SandboxContext`.
1. Obtain the `BackendDescriptor` and trusted `SandboxContext`.
2. Verify the descriptor and resolve its `backend_name` and version without fallback.
3. Call `attach` to obtain `Bound`.
4. Connect the boundary's `NetworkMediationSource` to network mediation.
5. Call `confirm` to obtain `Ready`, then `start_agent` to obtain `Running`.
6. Use the returned runtime handles for agent wait, `exec`, and port forwarding while network mediation consumes outbound connections.
This RFC supersedes RFC 0001's fixed in-sandbox supervisor placement and its assignment of topology-specific isolation controls to that process, generalizing the supervisor into a logical role. RFC 0001's authentication, sandbox-identity, outbound-connection, session, and reconnection requirements continue to apply. The component hosting the logical supervisor holds the required outbound gateway connection. A driver-hosted or shared supervisor routes gateway `exec`, SSH, and forwarding requests through the backend; the gateway does not initiate a connection to the boundary.
This RFC supersedes RFC 0001's fixed in-sandbox supervisor placement and its assignment of driver-specific isolation controls to that process, generalizing the supervisor into a logical role. RFC 0001's authentication, sandbox-identity, outbound-connection, session, and reconnection requirements continue to apply. The component hosting the logical supervisor holds the required outbound gateway connection. A driver-hosted or shared supervisor routes gateway `exec`, SSH, and forwarding requests through the backend; the gateway does not initiate a connection to the boundary.
### Failure semantics
@@ -327,65 +327,65 @@ enum BackendErrorKind { Invalid, Denied, Unavailable, Unsupported, Failed, Termi
`Invalid` covers descriptor, version, and backend mismatches; `Denied` covers authenticated attachment rejection; `Unavailable` covers transient inability to serve an operation; `Unsupported` identifies an optional operation the selected backend does not implement; `Failed` covers other backend faults; and `Terminated` reports boundary or workload termination, or an operation against an inactive boundary. An error never advances the lifecycle or authorizes an operation, and backend selection never falls back.
A backend may retry backend-private work within one `attach` call. The supervisor calls `attach` at most once per provisioned topology. If it does not return `Bound`, the topology is reclaimed rather than reused.
A backend may retry backend-private work within one `attach` call. The supervisor calls `attach` at most once per provisioned sandbox. If it does not return `Bound`, the sandbox is reclaimed rather than reused.
Failures resolve as follows:
- an `attach` or `confirm` failure, or network-mediation initialization failure while `Bound`, prevents untrusted workload execution and causes the driver to reclaim the topology;
- if `start_agent` does not return `Running`, no untrusted process from that attempt remains, and the driver reclaims the topology;
- an `attach` or `confirm` failure, or network-mediation initialization failure while `Bound`, prevents untrusted workload execution and causes the driver to reclaim the sandbox;
- if `start_agent` does not return `Running`, no untrusted process from that attempt remains, and the driver reclaims the sandbox;
- if `exec` or loopback `connect` fails, the backend terminates any process or closes any connection created by that attempt while the boundary otherwise remains active;
- after `Running`, supervisor or enforcement loss follows invariant 6; when enforcement loss ends the agent, `BoundaryProcess::wait` fails with `BackendErrorKind::Terminated` where process-exit observation survives;
- network-mediation errors yield no authorized connection and do not by themselves end `Running`; and
- retained runtime handles and the network-mediation source reject new operations whenever the boundary ends, except `BoundaryProcess::wait` where the backend can still return its stable result.
Whenever a boundary ends, the backend terminates remaining workload processes and releases the active-boundary binding before the compute driver reclaims or deprovisions the topology. If normal backend cleanup is unavailable, the compute driver uses the topology's trusted cleanup path to terminate the execution environment and invalidate the binding before reclaim or reuse. On normal agent exit, `BoundaryProcess::wait` returns the stable exit status. A retained `wait` result may outlive teardown.
Whenever a boundary ends, the backend terminates remaining workload processes and releases the active-boundary binding before the compute driver reclaims or deprovisions the sandbox. If normal backend cleanup is unavailable, the compute driver uses its trusted cleanup path to terminate the execution environment and invalidate the binding before reclaim or reuse. On normal agent exit, `BoundaryProcess::wait` returns the stable exit status. A retained `wait` result may outlive teardown.
### Topologies
### Compute-driver placement
The contract fixes the roles; a topology fixes their placement. Components may be co-located with the workload or hosted in trusted services, and one component may implement multiple roles. Every arrangement preserves the same lifecycle, interfaces, and invariants. Actual containment depends on the workload's kernel relationship to the trusted components. The non-normative [topology matrix](./topology-matrix.md) catalogs representative placements.
The contract fixes the roles; each compute driver decides where they run. Components may be co-located with the workload or hosted in trusted services, and one component may implement multiple roles. Every arrangement preserves the same lifecycle, interfaces, and invariants. Actual containment depends on the workload's kernel relationship to the trusted components. The non-normative [driver placement guide](./driver-placement.md) describes the supported mappings.
## Conformance
Every admitted backend and topology must pass tests for the six contract invariants plus descriptor verification, lifecycle ordering, runtime operations, and failure semantics.
Every admitted backend and compute driver must pass tests for the six contract invariants plus descriptor verification, lifecycle ordering, runtime operations, and failure semantics.
The conformance suite exercises each backend in its deployed topology, including delegated components and their transport. It verifies observable contract behavior while transport details remain backend-private.
The conformance suite exercises each backend in its driver-orchestrated placement, including delegated components and their transport. It verifies observable contract behavior while transport details remain backend-private.
## Risks
| Risk | Mitigation |
|---|---|
| The Isolation Backend could duplicate compute-driver responsibilities or allow topology-specific behavior to leak back into the supervisor. | Keep the responsibility boundary explicit: the compute driver owns, provisions, and deprovisions the topology; the backend binds and operates the active boundary. The same component may implement both roles. |
| Contract conformance could be mistaken for equivalent isolation across topologies. | Treat conformance as behavioral, not as a security-strength rating. Document and validate each topology's actual containment and reject policy it cannot enforce. |
| The Isolation Backend could duplicate compute-driver responsibilities or allow driver-specific behavior to leak back into the supervisor. | Keep the responsibility boundary explicit: the compute driver orchestrates and deprovisions the sandbox infrastructure; the backend binds and operates the active boundary. The same component may implement both roles. |
| Contract conformance could be mistaken for equivalent isolation across compute drivers. | Treat conformance as behavioral, not as a security-strength rating. Document and validate each driver's actual containment and reject policy it cannot enforce. |
| Shared backend or network-mediation components concentrate privilege and failure impact. | Isolate state, connection attribution, enforcement, and control authority per boundary. Failure of one boundary must not weaken another or enable direct egress. |
| The mandatory contract may exclude otherwise useful but incomplete backends. | Keep the network-mediation source, binary identity, process control, `exec`, and port forwarding mandatory. An incomplete backend does not claim conformance or silently degrade. |
| A future topology may not fit the lifecycle or interfaces. | Keep placement and coordination backend-private. Add versioned contract surface only when a concrete implementation requires new common semantics. |
| A future runtime may not fit the lifecycle or interfaces. | Keep placement and coordination backend-private. Add versioned contract surface only when a concrete implementation requires new common semantics. |
| A component restart may interrupt boundary operation. | Keep the last confirmed enforcement state in force and deny supervisor-dependent operations. |
## Alternatives
### Keep isolation embedded in the supervisor
OpenShell could keep the current in-pod design and add topology-specific supervisor and compute-driver paths as new requirements arise.
OpenShell could keep the current in-pod design and add driver-specific supervisor paths as new requirements arise.
Doing nothing avoids a new interface, but retains privileged boundary construction beside the workload. Implementing each delegated topology as a one-off supervisor change moves that privilege for one placement but accretes topology-specific supervisor behavior. The proposed contract instead keeps one supervisor lifecycle while allowing the topology to change.
Doing nothing avoids a new interface, but retains privileged boundary construction beside the workload. Implementing each runtime placement as a one-off supervisor change moves that privilege for one driver but adds driver-specific supervisor behavior. The proposed contract instead keeps one supervisor lifecycle while allowing runtime implementations and placements to change.
### Extend the compute-driver contract
The compute driver could own both provisioning and active-boundary operation.
This is natural for topologies such as MXC, and the same component may implement both responsibilities. The interfaces remain distinct because they serve different callers and lifecycles: the gateway uses the compute driver to provision and deprovision resources, while the supervisor uses the Isolation Backend to operate an active boundary. Combining them would couple runtime policy, identity, network mediation, and process operations to the gateway-facing driver API.
This is natural for drivers such as MXC, and the same component may implement both responsibilities. The interfaces remain distinct because they serve different callers and lifecycles: the gateway uses the compute driver to provision and deprovision resources, while the supervisor uses the Isolation Backend to operate an active boundary. Combining them would couple runtime policy, identity, network mediation, and process operations to the gateway-facing driver API.
### Start with a remote backend service
The contract could be expressed as a gRPC service or plugin ABI rather than an in-process Rust contract. [RFC 0001](../0001-core-architecture/README.md) chose gRPC for its gateway-facing drivers, so the question applies here.
The callers differ. A gateway driver is a control-plane peer with its own release cycle, while the Isolation Backend is driven by the supervisor that operates the boundary, and the co-located topology needs no transport at all. Starting in-process serves that case directly and lets delegated implementations carry their own transport behind the same interface. A transport-bearing surface is not precluded: it is versioned contract surface, added when a concrete delegated backend requires it.
The callers differ. A gateway driver is a control-plane peer with its own release cycle, while the Isolation Backend is driven by the supervisor that operates the boundary, and a co-located implementation needs no transport at all. Starting in-process serves that case directly and lets delegated implementations carry their own transport behind the same interface. A transport-bearing surface is not precluded: it is versioned contract surface, added when a concrete delegated backend requires it.
### Standardize topology and capabilities
### Standardize placement and capabilities
The contract could expose common topology roles, placement fields, capability flags, and recovery behavior so the supervisor can compose backend components.
The contract could expose common placement roles, capability flags, and recovery behavior so the supervisor can compose backend components.
That would make known deployments explicit, but it would also encode current topology assumptions and introduce capability-dependent supervisor paths. The proposal keeps placement and coordination in the opaque descriptor, requires one baseline contract, and uses the non-normative topology matrix to document representative arrangements.
That would make known deployments explicit, but it would also encode current placement assumptions and introduce capability-dependent supervisor paths. The proposal keeps placement and coordination in the opaque descriptor, requires one baseline contract, and uses the non-normative driver placement guide to document supported arrangements.
## Prior art
@@ -8,9 +8,9 @@ patterns locate the same code on newer revisions.
| Claim | Reference |
|---|---|
| Combined-topology agent container's seven capabilities | `crates/openshell-driver-kubernetes/src/driver.rs:2538` (base `SYS_ADMIN`/`NET_ADMIN`/`SYS_PTRACE`/`SYSLOG`), `:2544` (`SETUID`/`SETGID`/`DAC_READ_SEARCH` under userns). `rg -n -e SYS_ADMIN -e NET_ADMIN -e SYS_PTRACE -e SYSLOG -e SETUID -e SETGID -e DAC_READ_SEARCH crates/openshell-driver-kubernetes/src/driver.rs` |
| Legacy agent container's seven capabilities | `crates/openshell-driver-kubernetes/src/driver.rs:2538` (base `SYS_ADMIN`/`NET_ADMIN`/`SYS_PTRACE`/`SYSLOG`), `:2544` (`SETUID`/`SETGID`/`DAC_READ_SEARCH` under userns). `rg -n -e SYS_ADMIN -e NET_ADMIN -e SYS_PTRACE -e SYSLOG -e SETUID -e SETGID -e DAC_READ_SEARCH crates/openshell-driver-kubernetes/src/driver.rs` |
| Spec already separated from the netns handle | `crates/openshell-supervisor-process/src/process.rs:527`/`:535` (`ProcessHandle::spawn` takes `netns: Option<&NetworkNamespace>`) |
| Six `setns(CLONE_NEWNET)` call sites the contract's runtime interfaces replace (agent launch, SSH exec and forward, supervisor sessions, and namespace construction or entry); plus `CLONE_NEWNS` at `:449` (`unshare`, private mount namespace) and `:480` (`setns`, enter mount namespace, added for sidecar topology) | `process.rs:695`, `ssh.rs:653`/`:1262`, `supervisor_session.rs:735`, `netns/mod.rs:226`/`:342` (`rg -n "CLONE_NEWNET" crates/openshell-supervisor-process`) |
| Six `setns(CLONE_NEWNET)` call sites the contract's runtime interfaces replace (agent launch, SSH exec and forward, supervisor sessions, and namespace construction or entry); plus `CLONE_NEWNS` at `:449` (`unshare`, private mount namespace) and `:480` (`setns`, enter mount namespace, added for the legacy sidecar placement) | `process.rs:695`, `ssh.rs:653`/`:1262`, `supervisor_session.rs:735`, `netns/mod.rs:226`/`:342` (`rg -n "CLONE_NEWNET" crates/openshell-supervisor-process`) |
| `nft`-absent fail-open (the invariant bug), in-pod path only; the sidecar path uses `nft` with an `iptables-legacy` fallback and returns an error if neither establishes enforcement | `crates/openshell-supervisor-process/src/netns/mod.rs:265`; logs and returns `Ok(())` at `:277`; sidecar fallback at `:459`-`:471` |
| In-pod nftables ceiling is accept-by-default and rejects only TCP and UDP, so reading it back does not prove "only the proxy can egress" | `crates/openshell-supervisor-process/src/netns/nft_ruleset.rs:53` (`type filter hook output priority 0; policy accept`), `:56`-`:92` (proxy/loopback/established accept, then `reject` for IPv4 and IPv6 TCP and UDP only at `:106`+; other protocols and raw sockets pass once the host forwards the subnet). `rg -n "policy accept" crates/openshell-supervisor-process/src/netns` |
| Compute driver owns the execution domain (cgroup/resources, security context, device allocation set on the pod by the driver, not the supervisor) | `crates/openshell-driver-kubernetes/src/driver.rs` builds the pod/container spec; `rg -n -e securityContext -e resources -e 'cdi\.k8s\.io' -e devices crates/openshell-driver-kubernetes/src` |
@@ -18,9 +18,9 @@ patterns locate the same code on newer revisions.
| No `StartSandbox` RPC (create and start fused; no driver start gate) | `proto/compute_driver.proto` has `CreateSandbox`/`StopSandbox`/`DeleteSandbox` only |
| Gateway already speaks exec/session/port-forward; no lifecycle `Attach` (`AttachSandboxProvider` exists but attaches a provider record to a running sandbox, not the isolation lifecycle) | `proto/openshell.proto` (`ExecSandbox`, `ExecSandboxInteractive`, `CreateSshSession`, `ForwardTcp`, `AttachSandboxProvider`) |
| Agent command via CLI/`SANDBOX_COMMAND`; no admission-bound spec field today; the `sleep infinity` placeholder resolves `sleep` from the agent image's own filesystem | `crates/openshell-sandbox/src/main.rs:601`; K8s driver sets `sleep infinity` via `SANDBOX_COMMAND` at `driver.rs:2937` (`rg -n "sleep infinity" crates/openshell-driver-kubernetes/src`) |
| Init containers: `copy-self` (trusted, the OpenShell binary) and `workspace-init` (runs as root from the agent's own image, so its executables are image-provided); sidecar topology adds `openshell-network-init` (nftables setup, `NET_ADMIN`/`NET_RAW`/`CHOWN`/`FOWNER`) and `openshell-supervisor-network` runtime sidecar | `driver.rs:423` (`WORKSPACE_INIT_CONTAINER_NAME`), `:1506` (`copy-self` invocation), `:2113` (workspace-init container), `:1423` (`SUPERVISOR_NETWORK_INIT_CONTAINER_NAME`), `:1426` (`SUPERVISOR_NETWORK_SIDECAR_NAME`); `rg -n -e restart_policy -e workspace-init -e openshell-network-init -e openshell-supervisor-network crates/openshell-driver-kubernetes/src` |
| Init containers: `copy-self` (trusted, the OpenShell binary) and `workspace-init` (runs as root from the agent's own image, so its executables are image-provided); the legacy sidecar placement adds `openshell-network-init` (nftables setup, `NET_ADMIN`/`NET_RAW`/`CHOWN`/`FOWNER`) and `openshell-supervisor-network` | `driver.rs:423` (`WORKSPACE_INIT_CONTAINER_NAME`), `:1506` (`copy-self` invocation), `:2113` (workspace-init container), `:1423` (`SUPERVISOR_NETWORK_INIT_CONTAINER_NAME`), `:1426` (`SUPERVISOR_NETWORK_SIDECAR_NAME`); `rg -n -e restart_policy -e workspace-init -e openshell-network-init -e openshell-supervisor-network crates/openshell-driver-kubernetes/src` |
| Network policy is OPA per-CONNECT, not the boundary; identity via procfs | `crates/openshell-supervisor-network/src/opa.rs` (`NetworkInput`: `binary_path`/`binary_sha256`/`ancestors`/`cmdline_paths`), `procfs.rs`, glued in `proxy.rs:1955` (`authorize_egress_intent`; `NetworkInput` built at `:2032`) |
| Network enforcement already shares one implementation across placements: the combined path constructs and retains networking before agent launch, while the network-only sidecar owns the proxy, policy polling, and a topology-private control channel | `crates/openshell-sandbox/src/lib.rs:355` (`networking`), `:389` (`sidecar_control_server`), `:590` (`run_policy_poll_loop`), `:686` (`run_process`, after networking setup), `:736` (network-only sidecar lifecycle). `rg -n -e 'let networking' -e sidecar_control_server -e run_policy_poll_loop -e 'Network-only sidecar mode' crates/openshell-sandbox/src/lib.rs` |
| Network enforcement already shares one implementation across legacy placements: the co-located path constructs and retains networking before agent launch, while the network-only sidecar owns the proxy, policy polling, and a private control channel | `crates/openshell-sandbox/src/lib.rs:355` (`networking`), `:389` (`sidecar_control_server`), `:590` (`run_policy_poll_loop`), `:686` (`run_process`, after networking setup), `:736` (network-only sidecar lifecycle). `rg -n -e 'let networking' -e sidecar_control_server -e run_policy_poll_loop -e 'Network-only sidecar mode' crates/openshell-sandbox/src/lib.rs` |
| Binary identity is resolved after the connection is accepted (`/proc/net/tcp` inode lookup, socket-owner search, `/proc/<pid>/exe`, then PPID walking), so executable identity and ancestry describe state observed through trusted kernel interfaces during policy evaluation rather than an atomic snapshot at `connect()` | `crates/openshell-supervisor-network/src/procfs.rs:165` (`resolve_tcp_peer_binary`), `:343` (`parse_proc_net_tcp`), `:441` (`find_socket_inode_owners`), `:227` (`read_ppid`), `:243` (`collect_ancestor_binaries`). `rg -n -e resolve_tcp_peer_binary -e collect_ancestor_binaries crates/openshell-supervisor-network/src` |
| Identity display paths come from reading `/proc/<pid>/exe`, and the digest covers that live executable object at resolution time rather than a re-read of the display path | `crates/openshell-supervisor-network/src/procfs.rs:127` (`binary_path`), `:134`-`:135` (`read_link` of `/proc/<pid>/exe`), `proxy.rs:1800`/`:1816` (binary and ancestors both via `verify_or_cache_process_exe`), `identity.rs:107` (hashes `/proc/<pid>/exe`); digest rationale at `procfs.rs:117`-`:122`. `rg -n -e 'fn binary_path' -e verify_or_cache_process_exe crates/openshell-supervisor-network/src` |
| Static privilege ceiling on every spawned process; OPA never evaluates exec | `process.rs:527` (`ProcessHandle::spawn`), `:710`/`:812` (`drop_privileges` call sites), `:721`/`:818`-`:819` (sandbox enforcement); SSH reaches the same `enter_netns_and_sandbox` path (`ssh.rs:1245`) |
@@ -0,0 +1,41 @@
# Compute-driver placement
This non-normative guide shows how each compute driver places RFC 0012's
components and protects the supervisor-to-sandbox channel. Placement is a
driver concern; every driver presents the same Isolation Backend interface to
the supervisor.
| Compute driver | Supervisor | Sandbox runtime | Protected channel | Outer egress fence | Status |
|---|---|---|---|---|---|
| **Docker** | Host process | `openshell-sandbox` in the workload container | Authenticated Unix-domain socket | `network_mode=none` | Implemented in #2965 |
| **Podman** | Host process | `openshell-sandbox` in the workload container | Authenticated Unix-domain socket | `--network=none` | Implemented in #3230 |
| **Kubernetes** | Separate trusted Pod | `openshell-sandbox` in the workload Pod | Authenticated TLS over a private Service | `NetworkPolicy` | Implemented in #3144; requires a conforming NetworkPolicy CNI and trusted namespace |
| **VM** | Host process | `openshell-sandbox` as guest PID 1 | Authenticated vsock | No guest NIC | Implemented in #2945 |
The Kubernetes driver creates the workload fence, pair labels, private Service,
supervisor workload, immutable bootstrap Secret, and stable
namespace/Sandbox/workload/NetworkPolicy claims. The workload Pod has no direct
egress. DNS and TCP requests cross the protected channel, and the supervisor
initiates approved external connections. Admission requires a conforming CNI
and a namespace where untrusted principals cannot create Pods, mutate pair
labels, or read bootstrap Secrets. Pod readiness or the existence of a
`NetworkPolicy` object alone does not prove enforcement.
## Durable rules
- Every active boundary has one verified descriptor, one trusted
`SandboxContext`, one supervisor, and one sandbox runtime.
- Physical processes and listeners may be shared, but lifecycle state, policy,
binary identity, enforcement, and cleanup remain isolated per boundary.
- Moving a privileged component does not itself provide kernel separation.
## Kernel relationships
| Relationship | Meaning |
|---|---|
| **Shared host kernel** | The workload and the trusted components relied on for containment run on the host's kernel. |
| **Shared guest kernel** | The workload and those trusted components share one isolated VM guest kernel. |
| **Kernel-separated** | The trusted components relied on for containment run outside the workload's kernel. |
This guide is non-normative. It illustrates implementations of RFC 0012; it
does not extend the contract.
@@ -1,47 +0,0 @@
# Topology matrix
This non-normative matrix compares representative mappings of RFC 0012's logical
roles. It records role placement, sharing, and relationship to the workload
kernel; it does not select a deployment or establish conformance.
## Representative placements
| Pattern | Control and network-mediation placement | Boundary-mode placement | Workload-kernel relationship | Topology status |
|---|---|---|---|---|
| **Co-located/in-pod** | With the workload; the legacy in-process backend may omit boundary mode | Same process when used | Trusted components share the workload's host kernel | Placement implemented (original topology) |
| **Kubernetes proxy pod** | Trusted control pod | Boundary-mode workload entrypoint owning the workload PID and network namespaces | Shared cluster-node kernel; pod security boundaries separate control from workload | Implemented in #3144; requires a conforming NetworkPolicy CNI and trusted namespace |
| **Docker** | Gateway host | Trusted container entrypoint sharing the workload container's PID and network namespaces | Shared host kernel | Implemented in #2965 |
| **MicroVM** | Gateway host | Guest PID 1 | Boundary mode shares the guest kernel; control is kernel-separated | Implemented in #2945 |
The Kubernetes proxy-pod topology uses the same boundary protocol as Docker and
VM, with per-boundary TLS because the connection traverses the pod network.
Kubernetes-specific code provisions the workload fence, pair labels, boundary
Service, control Deployment, immutable bootstrap Secret, and stable
namespace/Sandbox/Deployment/NetworkPolicy claims. The workload pod has no
direct egress; attributed proxy streams cross the TLS channel and hostname
resolution occurs on the control side. Admission requires an explicitly acknowledged conforming CNI and a
namespace in which untrusted principals cannot create pods, mutate pair labels,
or read bootstrap Secrets. Pod readiness or the existence of a `NetworkPolicy`
object alone does not prove enforcement.
## Durable rules
- Every active boundary has one verified descriptor, one trusted
`SandboxContext`, one control role, and at most one boundary role. Those
processes form one logical supervisor.
- Physical processes and listeners may be shared, but lifecycle state, policy,
binary identity, enforcement, and cleanup remain isolated per boundary.
- Moving a privileged component does not itself provide kernel separation.
## Kernel relationships
| Relationship | Meaning |
|---|---|
| **Shared host kernel** | The workload and the trusted components relied on for containment run on the host's kernel. |
| **Shared guest or application kernel** | The workload and those trusted components share one isolated kernel: a VM guest kernel or a userspace application kernel. |
| **Kernel-separated** | The trusted components relied on for containment run outside the workload's kernel. |
## Status
This matrix is non-normative. It illustrates implementations of RFC 0012; it
does not extend the contract.
+8 -8
View File
@@ -173,7 +173,7 @@ rationale, configured and effective modes, active generation, and the explicit
docker info
docker ps --filter name=openshell
docker logs <container> --tail=200
docker run --rm --entrypoint /openshell-sandbox "${OPENSHELL_DOCKER_SUPERVISOR_IMAGE:-ghcr.io/nvidia/openshell/supervisor:latest}" --version
docker run --rm --entrypoint /openshell-sandbox "${OPENSHELL_SANDBOX_RUNTIME_IMAGE:-ghcr.io/nvidia/openshell/sandbox:latest}" --version
openshell status
```
@@ -213,7 +213,7 @@ Common findings:
- Sandbox fails before readiness with an OCI workspace validation error: inspect the image's `WorkingDir` using the immutable image ID reported by the gateway. Empty, `/`, and explicit `/sandbox` use the managed `/sandbox` compatibility workspace. Any other workdir must be an absolute normalized directory with no symlink components; the final policy UID, primary GID, and supplementary groups must pass the kernel's effective traverse/write checks, including POSIX ACL and LSM decisions. OpenShell does not create, chown, or chmod a non-default image workdir.
- Docker also rejects an image `VOLUME` that covers the workdir or one of its parents because the runtime would mask the immutable path before validation. Move the `VOLUME` below the workspace or remove the declaration.
- A workdir rejected as a special filesystem or OpenShell control-path collision cannot be made valid with permissions. Move the image workdir away from kernel-backed mounts and the concrete supervisor, TLS, token, runtime, and socket paths named in the error.
- Docker driver cannot initialize because it cannot find `openshell-sandbox`: verify `OPENSHELL_DOCKER_SUPERVISOR_BIN`, the sibling binary next to `openshell-gateway`, or the configured supervisor image contains `/openshell-sandbox`.
- Docker driver cannot initialize because it cannot find `openshell-sandbox`: verify the sibling binary next to `openshell-gateway`, or that the configured `sandbox_runtime_image` contains `/openshell-sandbox`.
- Sandbox never registers: check gateway logs and supervisor callback endpoint.
- On macOS, repeated `Policy fetch failed after 5 attempts` messages with a
Homebrew gateway bound to `[::1]:17670` indicate that the Docker
@@ -222,7 +222,7 @@ Common findings:
`127.0.0.1:17670` primary listener, and reuse it for authenticated sandbox
callbacks. On an older release, set `bind_address = "127.0.0.1:17670"` or
upgrade.
- Supervisor image exits before printing `openshell-sandbox --version`: verify the configured supervisor image contains a static executable at `/openshell-sandbox`.
- Sandbox runtime image exits before printing `openshell-sandbox --version`: verify the configured image contains a static executable at `/openshell-sandbox`.
- A sandbox with explicit `protocol: tcp` endpoints fails before workload readiness: confirm the Docker or Podman driver supplied the `policy-dns-transparent-tcp` runtime capability and inspect supervisor logs for missing `nft`, synthetic-route overlap, or namespace-local DNS/TCP listener bind failures. Kubernetes, VM, sidecar, and out-of-tree drivers must reject this policy until they provide the complete substrate; use omitted protocol with an explicit proxy on those runtimes.
- A GPU sandbox fails because Docker reports no discovered NVIDIA CDI devices: verify `.DiscoveredDevices` contains entries such as `nvidia.com/gpu=all`, verify `/etc/cdi` or `/var/run/cdi` contains a generated NVIDIA spec, and check that `nvidia-cdi-refresh.service` and `nvidia-cdi-refresh.path` from NVIDIA Container Toolkit are enabled and healthy. The service is a one-shot unit, so `inactive (dead)` can be normal after a successful run; use `systemctl status` and `journalctl` to distinguish success from a skipped or failed refresh. Restart `nvidia-cdi-refresh.service` to regenerate missing or stale CDI specs, then restart or reload Docker and re-check `docker info`.
@@ -269,12 +269,12 @@ Common findings:
When `userns` is configured (e.g. `userns = "auto"` or `userns = "keep-id"`):
- Supervisor delivery uses bind-mount fallback instead of image volumes because
- Sandbox runtime delivery uses bind-mount fallback instead of image volumes because
overlay mounts do not support `idmapped` mounts. The supervisor binary is
extracted from the supervisor image and cached at
extracted from the sandbox runtime image and cached at
`$XDG_DATA_HOME/openshell/podman-supervisor/` (typically
`~/.local/share/openshell/podman-supervisor/`).
- Stale cache: if the supervisor image is updated but the cached binary is not
- Stale cache: if the sandbox runtime image is updated but the cached binary is not
refreshed, sandbox creation may fail with an ELF validation error or version
mismatch. Remove the cache directory and retry.
- `auto` mode requires subuid/subgid ranges for the current user in
@@ -454,7 +454,7 @@ kubectl -n openshell get statefulset openshell -o jsonpath="{.spec.template.spec
helm -n openshell get values openshell | grep -E 'repository|tag|supervisorImage|workload'
```
The gateway and supervisor images should use the same release tag. A stale supervisor image can make sandbox behavior lag behind gateway policy or protocol changes.
The gateway, sandbox, and supervisor images should use the same release tag. A stale runtime image can make sandbox behavior lag behind gateway policy or protocol changes.
For vulnerability reports, record the running image digest and scan that exact
artifact. The gateway includes a pinned Distroless base; the supervisor includes
@@ -733,7 +733,7 @@ configuration — check that the gateway spawned the driver binary you expect
| `openshell status` fails | Gateway endpoint unreachable or auth mismatch | `openshell gateway info`, gateway logs |
| `BatchSpanProcessor.ExportError` repeatedly reports connection refused on `127.0.0.1:4317` | The local gateway started with OTLP configured but the collector forwarding task later stopped, or the config was created manually | Restart `gateway:docker`, `gateway:podman`, or `gateway:vm` so it re-detects the listener; inspect the generated `gateway.toml` for `[openshell.gateway.otlp]` |
| Gateway starts but sandbox create fails | Compute driver cannot reach runtime | Docker/Podman/Kubernetes/VM driver logs |
| Gateway exits while resolving compute-driver listener requirements | Callback alias topology is unsupported, the Podman network cannot be inspected, or the selected address is not private/authorized | Gateway startup error, `podman info --debug`, Podman network inspection, host IPv4 default route |
| Gateway exits while resolving compute-driver listener requirements | The callback hostname is unsupported, the Podman network cannot be inspected, or the selected address is not private/authorized | Gateway startup error, `podman info --debug`, Podman network inspection, host IPv4 default route |
| Admin, health, reflection, or HTTP request is denied on an additional Docker/Podman callback-only listener | Additional callback listeners intentionally expose only sandbox-callable gRPC methods | Retry through the gateway's primary endpoint; inspect the listener-purpose startup log if the address was unexpected |
| Docker or Podman sandbox never registers | Wrong callback endpoint or supervisor startup failure | Gateway logs and sandbox container logs |
| Docker GPU sandbox fails before startup | NVIDIA CDI specs are missing or Docker has not discovered them | `docker info --format '{{json .DiscoveredDevices}}'`, `/etc/cdi`, `/var/run/cdi`, `nvidia-cdi-refresh.service` |