mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-28 07:13:04 +08:00
feat(docker): select supervisor networking for ECI workloads
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
This commit is contained in:
@@ -41,9 +41,11 @@ mediates every supported TCP and DNS operation, attributes it to the calling
|
||||
binary, and sends the request across the private channel. The supervisor
|
||||
authorizes the request before it opens an upstream connection. Docker's absent
|
||||
workload network is the mandatory outer fence if mediation fails or is
|
||||
bypassed. The trusted supervisor companion uses Docker host networking, where
|
||||
it reaches the gateway's primary loopback listener and originates approved
|
||||
egress.
|
||||
bypassed. The trusted supervisor companion originates approved egress. Its
|
||||
operator-owned `supervisor_network_mode` defaults to `auto`: each launch
|
||||
inspects the existing workload container and chooses bridge for `sysbox-runc`
|
||||
(the Docker Desktop ECI runtime), or host for other runtimes. Explicit `host`
|
||||
and `bridge` values override this heuristic. No probe container is created.
|
||||
|
||||
The driver copies trusted runtime bytes from the configured supervisor image
|
||||
through the Docker archive API. No workload launch depends on a host bind
|
||||
@@ -158,7 +160,7 @@ openshell sandbox create \
|
||||
`/openshell-sandbox` binary. The driver extracts that binary as bytes and
|
||||
stages it into the stopped workload. `supervisor_image` contains the
|
||||
dynamically linked glibc `/openshell-supervisor` binary that runs in the
|
||||
host-networked supervisor container. Release and gateway image builds bake
|
||||
supervisor companion container. Release and gateway image builds bake
|
||||
matching image tags into the binary.
|
||||
|
||||
## Gateway session and TLS
|
||||
@@ -172,7 +174,7 @@ When no endpoint is configured, the supervisor connects to
|
||||
Docker daemon host. A configured HTTPS server certificate must include the
|
||||
endpoint host in its subject alternative names.
|
||||
|
||||
The driver publishes host loopback as the backend address for
|
||||
In host mode, the driver publishes host loopback as the backend address for
|
||||
`host.openshell.internal`. Policy DNS resolves that reserved name through the
|
||||
mediated path, so policies can reach host services without a Docker bridge,
|
||||
container DNS alias, or another gateway listener.
|
||||
@@ -181,9 +183,26 @@ For HTTPS endpoints, the supervisor receives only the gateway CA and
|
||||
uses its sandbox-scoped bearer token to authenticate RPCs. User client
|
||||
certificates and private keys are not delivered to either container.
|
||||
|
||||
Docker Engine on Linux supports host networking directly. Docker Desktop
|
||||
requires host networking to be enabled in Settings and does not support it
|
||||
when Enhanced Container Isolation is enabled.
|
||||
Host mode on Docker Desktop requires host networking enabled in Settings and
|
||||
ECI disabled. Bridge mode uses Docker-resolved `host-gateway` entries for both
|
||||
host aliases. The supervisor's `--host-gateway-from-hosts` option pins the
|
||||
numeric address from its own driver-owned `/etc/hosts` into the runtime
|
||||
descriptor before attaching the workload; workload DNS and hosts files cannot
|
||||
select this address. Gateway endpoints and host aliases are independent in
|
||||
bridge mode.
|
||||
|
||||
When `grpc_endpoint` is omitted, bridge mode uses `host.docker.internal` on
|
||||
Desktop or for a wildcard gateway listener, and the concrete bind address for
|
||||
other non-loopback Linux listeners. The port is the gateway bind port; remapped
|
||||
container publications require an explicit endpoint. Native Linux loopback-only
|
||||
listeners cannot serve a bridge supervisor and produce an actionable error.
|
||||
Explicit endpoints retain TLS verification and are never rewritten; bridge
|
||||
rejects loopback/unspecified endpoint addresses. Host services bound only to
|
||||
loopback are also unreachable from a native Linux bridge supervisor.
|
||||
|
||||
Selecting bridge does not qualify full ECI support. Landlock and seccomp probes
|
||||
remain mandatory, and a gateway container under ECI needs an administrator's
|
||||
Docker socket exception.
|
||||
|
||||
The supervisor owns these security-critical variables:
|
||||
|
||||
|
||||
@@ -153,6 +153,17 @@ fn provisioning_span(
|
||||
span
|
||||
}
|
||||
|
||||
/// Operator-owned networking mode for the trusted supervisor companion.
|
||||
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum DockerSupervisorNetworkMode {
|
||||
/// Use bridge for Sysbox/ECI workloads, otherwise use host networking.
|
||||
#[default]
|
||||
Auto,
|
||||
Host,
|
||||
Bridge,
|
||||
}
|
||||
|
||||
/// Gateway-local configuration for the Docker compute driver.
|
||||
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
|
||||
#[serde(default, deny_unknown_fields)]
|
||||
@@ -182,6 +193,8 @@ pub struct DockerComputeConfig {
|
||||
/// supplies its own export endpoint; driver TOML cannot set it.
|
||||
#[serde(skip)]
|
||||
pub supervisor_otlp_endpoint: Option<String>,
|
||||
/// Networking mode for the supervisor only. Workloads always use network=none.
|
||||
pub supervisor_network_mode: DockerSupervisorNetworkMode,
|
||||
|
||||
/// Image containing the trusted `openshell-sandbox` binary.
|
||||
pub sandbox_runtime_image: Option<String>,
|
||||
@@ -282,6 +295,7 @@ impl Default for DockerComputeConfig {
|
||||
sandbox_label: "default".to_string(),
|
||||
grpc_endpoint: String::new(),
|
||||
supervisor_otlp_endpoint: None,
|
||||
supervisor_network_mode: DockerSupervisorNetworkMode::Auto,
|
||||
sandbox_runtime_image: None,
|
||||
supervisor_bin: None,
|
||||
supervisor_image: None,
|
||||
@@ -317,6 +331,9 @@ struct DockerDriverRuntimeConfig {
|
||||
supervisor_image_id: String,
|
||||
supervisor_grpc_endpoint: String,
|
||||
supervisor_otlp_endpoint: Option<String>,
|
||||
supervisor_network_mode: DockerSupervisorNetworkMode,
|
||||
supervisor_endpoint_explicit: bool,
|
||||
bridge_default_endpoint: Option<String>,
|
||||
ssh_socket_path: String,
|
||||
guest_tls: Option<DockerGuestTlsPaths>,
|
||||
gpu: DockerGpuRuntimeCapabilities,
|
||||
@@ -889,6 +906,14 @@ impl DockerComputeDriver {
|
||||
};
|
||||
validate_docker_app_armor_profile(docker_config.app_armor_profile.as_ref(), &info)?;
|
||||
let gateway_port = gateway_bind_address.port();
|
||||
let supervisor_endpoint_explicit = !docker_config.grpc_endpoint.trim().is_empty();
|
||||
let bridge_default_endpoint = default_docker_bridge_endpoint(
|
||||
gateway_bind_address,
|
||||
docker_guest_tls_configured(docker_config),
|
||||
info.operating_system
|
||||
.as_deref()
|
||||
.is_some_and(|os| os.to_ascii_lowercase().contains("docker desktop")),
|
||||
);
|
||||
let mut docker_config = docker_config.clone();
|
||||
if docker_config.grpc_endpoint.trim().is_empty() {
|
||||
docker_config.grpc_endpoint = default_docker_supervisor_grpc_endpoint(
|
||||
@@ -947,6 +972,9 @@ impl DockerComputeDriver {
|
||||
supervisor_image_id,
|
||||
supervisor_grpc_endpoint,
|
||||
supervisor_otlp_endpoint: docker_config.supervisor_otlp_endpoint.clone(),
|
||||
supervisor_network_mode: docker_config.supervisor_network_mode,
|
||||
supervisor_endpoint_explicit,
|
||||
bridge_default_endpoint,
|
||||
ssh_socket_path: docker_config.ssh_socket_path.clone(),
|
||||
guest_tls,
|
||||
gpu,
|
||||
@@ -4571,10 +4599,9 @@ async fn prepare_docker_boundary_files(
|
||||
server_name: tls.server_name.clone(),
|
||||
trust_anchor_pem: tls.trust_anchor_pem.clone(),
|
||||
},
|
||||
// Pin the reserved host alias to the same address used by the
|
||||
// host-networked supervisor. This is normally loopback, but container
|
||||
// CI reaches the gateway and host fixtures through the job
|
||||
// container's bridge address.
|
||||
// Host mode pins the alias to the gateway address, normally loopback.
|
||||
// In bridge mode the companion replaces this value from its own
|
||||
// Docker-injected hosts file before attaching the descriptor.
|
||||
host_gateway_ip: docker_supervisor_host_address(&config.supervisor_grpc_endpoint),
|
||||
workload_identity: workload_identity.clone(),
|
||||
child_env: docker_child_environment(sandbox),
|
||||
@@ -4981,14 +5008,34 @@ fn docker_auxiliary_container_labels(
|
||||
])
|
||||
}
|
||||
|
||||
fn docker_supervisor_host_config(mounts: Vec<Mount>, grpc_endpoint: &str) -> HostConfig {
|
||||
fn docker_supervisor_host_config(
|
||||
mounts: Vec<Mount>,
|
||||
grpc_endpoint: &str,
|
||||
network_mode: DockerSupervisorNetworkMode,
|
||||
) -> HostConfig {
|
||||
HostConfig {
|
||||
// The supervisor is trusted infrastructure and originates every
|
||||
// approved upstream connection. Host networking lets it reach the
|
||||
// configured gateway and host-side services directly; the workload
|
||||
// remains fenced by network=none.
|
||||
network_mode: Some("host".to_string()),
|
||||
extra_hosts: docker_supervisor_host_aliases(grpc_endpoint),
|
||||
network_mode: Some(
|
||||
if network_mode == DockerSupervisorNetworkMode::Bridge {
|
||||
"bridge"
|
||||
} else {
|
||||
"host"
|
||||
}
|
||||
.to_string(),
|
||||
),
|
||||
extra_hosts: if network_mode == DockerSupervisorNetworkMode::Bridge {
|
||||
// Docker resolves host-gateway on the daemon, including Desktop's VM.
|
||||
// The gateway endpoint may be remote and must not define host aliases.
|
||||
Some(vec![
|
||||
format!("{HOST_OPEN_SHELL_INTERNAL}:host-gateway"),
|
||||
format!("{HOST_DOCKER_INTERNAL}:host-gateway"),
|
||||
])
|
||||
} else {
|
||||
docker_supervisor_host_aliases(grpc_endpoint)
|
||||
},
|
||||
mounts: Some(mounts),
|
||||
cap_drop: Some(vec!["ALL".to_string()]),
|
||||
cap_add: None,
|
||||
@@ -5044,6 +5091,18 @@ async fn spawn_docker_control_process(
|
||||
config: &DockerDriverRuntimeConfig,
|
||||
failure_context: DockerRuntimeFailureContext,
|
||||
) -> Result<DockerControlProcess, Status> {
|
||||
// Inspect the existing workload on every supervisor launch, including restart.
|
||||
// This adds no diagnostic container and observes the actual selected runtime.
|
||||
let (network_mode, supervisor_endpoint) = Box::pin(async {
|
||||
let workload = docker
|
||||
.inspect_container(&failure_context.container_id, None)
|
||||
.await
|
||||
.map_err(|error| internal_status("inspect Docker workload runtime", error))?;
|
||||
docker_supervisor_networking(config, &workload)
|
||||
})
|
||||
.await?;
|
||||
info!(sandbox_id = %sandbox.id, ?network_mode, endpoint = %supervisor_endpoint,
|
||||
"Selected Docker supervisor networking");
|
||||
let directory = docker_boundary_state_dir(sandbox, config)?;
|
||||
let main_process_spec = tokio::fs::read_to_string(directory.join(MAIN_PROCESS_SPEC_FILE))
|
||||
.await
|
||||
@@ -5072,7 +5131,7 @@ async fn spawn_docker_control_process(
|
||||
format!(
|
||||
"{}={}",
|
||||
openshell_core::sandbox_env::ENDPOINT,
|
||||
config.supervisor_grpc_endpoint
|
||||
supervisor_endpoint
|
||||
),
|
||||
format!("{}={}", openshell_core::sandbox_env::SANDBOX_ID, sandbox.id),
|
||||
format!("{}={}", openshell_core::sandbox_env::SANDBOX, sandbox.name),
|
||||
@@ -5140,6 +5199,9 @@ async fn spawn_docker_control_process(
|
||||
workspace_root,
|
||||
format!("--health-socket-path={SUPERVISOR_HEALTH_SOCKET_PATH}"),
|
||||
];
|
||||
if network_mode == DockerSupervisorNetworkMode::Bridge {
|
||||
command.push("--host-gateway-from-hosts".to_string());
|
||||
}
|
||||
command.extend(docker_upstream_proxy_cli_args(
|
||||
&config.upstream_proxy,
|
||||
config.proxy_ca_bundle.is_some(),
|
||||
@@ -5203,7 +5265,8 @@ async fn spawn_docker_control_process(
|
||||
}),
|
||||
host_config: Some(docker_supervisor_host_config(
|
||||
supervisor_mounts,
|
||||
&config.supervisor_grpc_endpoint,
|
||||
&supervisor_endpoint,
|
||||
network_mode,
|
||||
)),
|
||||
..Default::default()
|
||||
};
|
||||
@@ -6556,6 +6619,63 @@ fn default_docker_supervisor_grpc_endpoint(gateway_port: u16, tls: bool) -> Stri
|
||||
format!("{scheme}://127.0.0.1:{gateway_port}")
|
||||
}
|
||||
|
||||
fn default_docker_bridge_endpoint(
|
||||
gateway_bind_address: SocketAddr,
|
||||
tls: bool,
|
||||
desktop: bool,
|
||||
) -> Option<String> {
|
||||
let scheme = if tls { "https" } else { "http" };
|
||||
if desktop || gateway_bind_address.ip().is_unspecified() {
|
||||
Some(format!(
|
||||
"{scheme}://{HOST_DOCKER_INTERNAL}:{}",
|
||||
gateway_bind_address.port()
|
||||
))
|
||||
} else if !gateway_bind_address.ip().is_loopback() {
|
||||
Some(format!("{scheme}://{gateway_bind_address}"))
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
fn docker_supervisor_networking(
|
||||
config: &DockerDriverRuntimeConfig,
|
||||
workload: &bollard::models::ContainerInspectResponse,
|
||||
) -> Result<(DockerSupervisorNetworkMode, String), Status> {
|
||||
let sysbox = workload
|
||||
.host_config
|
||||
.as_ref()
|
||||
.and_then(|host| host.runtime.as_deref())
|
||||
.is_some_and(|runtime| runtime == "sysbox-runc");
|
||||
let mode = match config.supervisor_network_mode {
|
||||
DockerSupervisorNetworkMode::Auto if sysbox => DockerSupervisorNetworkMode::Bridge,
|
||||
DockerSupervisorNetworkMode::Auto => DockerSupervisorNetworkMode::Host,
|
||||
mode => mode,
|
||||
};
|
||||
let endpoint = if mode == DockerSupervisorNetworkMode::Bridge
|
||||
&& !config.supervisor_endpoint_explicit
|
||||
{
|
||||
config.bridge_default_endpoint.clone().ok_or_else(|| Status::failed_precondition(
|
||||
"Docker bridge supervisor cannot reach a Linux gateway bound only to loopback; configure a bridge-reachable gateway bind_address and grpc_endpoint, or use supervisor_network_mode = 'host' when the runtime permits host networking",
|
||||
))?
|
||||
} else {
|
||||
config.supervisor_grpc_endpoint.clone()
|
||||
};
|
||||
if mode == DockerSupervisorNetworkMode::Bridge {
|
||||
let url = Url::parse(&endpoint).map_err(|error| {
|
||||
Status::failed_precondition(format!("invalid Docker supervisor endpoint: {error}"))
|
||||
})?;
|
||||
if matches!(url.host(), Some(url::Host::Domain(host)) if host.eq_ignore_ascii_case("localhost"))
|
||||
|| matches!(url.host(), Some(url::Host::Ipv4(ip)) if ip.is_loopback() || ip.is_unspecified())
|
||||
|| matches!(url.host(), Some(url::Host::Ipv6(ip)) if ip.is_loopback() || ip.is_unspecified())
|
||||
{
|
||||
return Err(Status::failed_precondition(
|
||||
"Docker bridge supervisor grpc_endpoint must be reachable outside the gateway's loopback namespace; use host.docker.internal or a reachable gateway address",
|
||||
));
|
||||
}
|
||||
}
|
||||
Ok((mode, endpoint))
|
||||
}
|
||||
|
||||
pub(crate) fn docker_guest_tls_paths(
|
||||
docker_config: &DockerComputeConfig,
|
||||
) -> CoreResult<Option<DockerGuestTlsPaths>> {
|
||||
|
||||
@@ -212,6 +212,9 @@ fn runtime_config() -> DockerDriverRuntimeConfig {
|
||||
supervisor_image_id: "sha256:supervisor-test".to_string(),
|
||||
supervisor_grpc_endpoint: "https://host.openshell.internal:8443".to_string(),
|
||||
supervisor_otlp_endpoint: None,
|
||||
supervisor_network_mode: DockerSupervisorNetworkMode::Auto,
|
||||
supervisor_endpoint_explicit: true,
|
||||
bridge_default_endpoint: Some("https://host.docker.internal:8443".to_string()),
|
||||
ssh_socket_path: openshell_core::container_paths::SSH_SOCKET_PATH.to_string(),
|
||||
guest_tls: Some(DockerGuestTlsPaths {
|
||||
ca: PathBuf::from("/tmp/ca.crt"),
|
||||
@@ -3030,7 +3033,11 @@ fn build_container_create_body_disables_docker_networking() {
|
||||
|
||||
#[test]
|
||||
fn docker_supervisor_uses_host_network() {
|
||||
let host = docker_supervisor_host_config(Vec::new(), "https://127.0.0.1:17670");
|
||||
let host = docker_supervisor_host_config(
|
||||
Vec::new(),
|
||||
"https://127.0.0.1:17670",
|
||||
DockerSupervisorNetworkMode::Host,
|
||||
);
|
||||
|
||||
assert_eq!(host.network_mode.as_deref(), Some("host"));
|
||||
assert_eq!(
|
||||
@@ -3046,7 +3053,11 @@ fn docker_supervisor_uses_host_network() {
|
||||
|
||||
#[test]
|
||||
fn docker_supervisor_maps_host_aliases_to_the_gateway_address() {
|
||||
let host = docker_supervisor_host_config(Vec::new(), "https://172.20.0.4:17670");
|
||||
let host = docker_supervisor_host_config(
|
||||
Vec::new(),
|
||||
"https://172.20.0.4:17670",
|
||||
DockerSupervisorNetworkMode::Host,
|
||||
);
|
||||
|
||||
assert_eq!(
|
||||
host.extra_hosts,
|
||||
@@ -3063,7 +3074,11 @@ fn docker_supervisor_maps_host_aliases_to_the_gateway_address() {
|
||||
|
||||
#[test]
|
||||
fn docker_supervisor_leaves_named_gateway_hosts_to_dns() {
|
||||
let host = docker_supervisor_host_config(Vec::new(), "https://gateway.example.com:17670");
|
||||
let host = docker_supervisor_host_config(
|
||||
Vec::new(),
|
||||
"https://gateway.example.com:17670",
|
||||
DockerSupervisorNetworkMode::Host,
|
||||
);
|
||||
|
||||
assert_eq!(host.extra_hosts, None);
|
||||
assert_eq!(
|
||||
@@ -3084,6 +3099,186 @@ fn docker_supervisor_defaults_to_the_primary_loopback_endpoint() {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn docker_supervisor_selects_bridge_for_sysbox_and_honors_operator_mode() {
|
||||
for (runtime, requested, expected) in [
|
||||
(
|
||||
Some("sysbox-runc"),
|
||||
DockerSupervisorNetworkMode::Auto,
|
||||
DockerSupervisorNetworkMode::Bridge,
|
||||
),
|
||||
(
|
||||
Some("runc"),
|
||||
DockerSupervisorNetworkMode::Auto,
|
||||
DockerSupervisorNetworkMode::Host,
|
||||
),
|
||||
(
|
||||
None,
|
||||
DockerSupervisorNetworkMode::Auto,
|
||||
DockerSupervisorNetworkMode::Host,
|
||||
),
|
||||
(
|
||||
Some("sysbox-runc"),
|
||||
DockerSupervisorNetworkMode::Host,
|
||||
DockerSupervisorNetworkMode::Host,
|
||||
),
|
||||
(
|
||||
Some("runc"),
|
||||
DockerSupervisorNetworkMode::Bridge,
|
||||
DockerSupervisorNetworkMode::Bridge,
|
||||
),
|
||||
] {
|
||||
let mut config = runtime_config();
|
||||
config.supervisor_network_mode = requested;
|
||||
config.supervisor_endpoint_explicit = false;
|
||||
config.supervisor_grpc_endpoint = "https://127.0.0.1:8443".into();
|
||||
let inspected = bollard::models::ContainerInspectResponse {
|
||||
host_config: Some(HostConfig {
|
||||
runtime: runtime.map(str::to_string),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
let (mode, endpoint) = docker_supervisor_networking(&config, &inspected).unwrap();
|
||||
assert_eq!(mode, expected);
|
||||
assert_eq!(
|
||||
endpoint,
|
||||
if mode == DockerSupervisorNetworkMode::Bridge {
|
||||
"https://host.docker.internal:8443"
|
||||
} else {
|
||||
"https://127.0.0.1:8443"
|
||||
}
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn docker_bridge_preserves_explicit_endpoints_and_rejects_loopback() {
|
||||
let mut config = runtime_config();
|
||||
config.supervisor_network_mode = DockerSupervisorNetworkMode::Bridge;
|
||||
for endpoint in [
|
||||
"https://gateway.example.com:9443",
|
||||
"http://172.20.0.4:8080",
|
||||
"https://[fd00::2]:9443",
|
||||
] {
|
||||
config.supervisor_grpc_endpoint = endpoint.into();
|
||||
let (_, selected) = docker_supervisor_networking(
|
||||
&config,
|
||||
&bollard::models::ContainerInspectResponse::default(),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(selected, endpoint);
|
||||
}
|
||||
for endpoint in [
|
||||
"https://127.0.0.1:8443",
|
||||
"http://localhost:8080",
|
||||
"https://[::1]:8443",
|
||||
"http://0.0.0.0:8080",
|
||||
"http://[::]:8080",
|
||||
] {
|
||||
config.supervisor_grpc_endpoint = endpoint.into();
|
||||
let error = docker_supervisor_networking(
|
||||
&config,
|
||||
&bollard::models::ContainerInspectResponse::default(),
|
||||
)
|
||||
.unwrap_err();
|
||||
assert_eq!(error.code(), tonic::Code::FailedPrecondition);
|
||||
assert!(error.message().contains("grpc_endpoint"));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn docker_bridge_defaults_respect_desktop_and_linux_listener_reachability() {
|
||||
assert_eq!(
|
||||
default_docker_bridge_endpoint("127.0.0.1:17670".parse().unwrap(), true, true).as_deref(),
|
||||
Some("https://host.docker.internal:17670")
|
||||
);
|
||||
assert_eq!(
|
||||
default_docker_bridge_endpoint("0.0.0.0:8080".parse().unwrap(), false, false).as_deref(),
|
||||
Some("http://host.docker.internal:8080")
|
||||
);
|
||||
assert_eq!(
|
||||
default_docker_bridge_endpoint("192.168.1.2:8080".parse().unwrap(), true, false).as_deref(),
|
||||
Some("https://192.168.1.2:8080")
|
||||
);
|
||||
assert_eq!(
|
||||
default_docker_bridge_endpoint("[fd00::2]:8080".parse().unwrap(), true, false).as_deref(),
|
||||
Some("https://[fd00::2]:8080")
|
||||
);
|
||||
assert!(
|
||||
default_docker_bridge_endpoint("127.0.0.1:17670".parse().unwrap(), true, false).is_none()
|
||||
);
|
||||
let mut config = runtime_config();
|
||||
config.supervisor_network_mode = DockerSupervisorNetworkMode::Bridge;
|
||||
config.supervisor_endpoint_explicit = false;
|
||||
config.bridge_default_endpoint = None;
|
||||
let error = docker_supervisor_networking(
|
||||
&config,
|
||||
&bollard::models::ContainerInspectResponse::default(),
|
||||
)
|
||||
.unwrap_err();
|
||||
assert!(error.message().contains("bind_address"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn docker_bridge_host_aliases_are_independent_of_gateway_endpoint() {
|
||||
let host = docker_supervisor_host_config(
|
||||
Vec::new(),
|
||||
"https://gateway.example.com:9443",
|
||||
DockerSupervisorNetworkMode::Bridge,
|
||||
);
|
||||
assert_eq!(host.network_mode.as_deref(), Some("bridge"));
|
||||
assert_eq!(
|
||||
host.extra_hosts,
|
||||
Some(vec![
|
||||
"host.openshell.internal:host-gateway".into(),
|
||||
"host.docker.internal:host-gateway".into()
|
||||
])
|
||||
);
|
||||
assert_eq!(host.cap_drop, Some(vec!["ALL".into()]));
|
||||
assert_eq!(host.cap_add, None);
|
||||
assert_eq!(
|
||||
host.security_opt,
|
||||
Some(vec!["no-new-privileges:true".into()])
|
||||
);
|
||||
assert_eq!(
|
||||
build_container_create_body(&test_sandbox(), &runtime_config())
|
||||
.unwrap()
|
||||
.host_config
|
||||
.unwrap()
|
||||
.network_mode
|
||||
.as_deref(),
|
||||
Some("none")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn docker_supervisor_network_mode_is_an_operator_config_enum() {
|
||||
assert_eq!(
|
||||
toml::from_str::<DockerComputeConfig>("")
|
||||
.unwrap()
|
||||
.supervisor_network_mode,
|
||||
DockerSupervisorNetworkMode::Auto
|
||||
);
|
||||
for (value, expected) in [
|
||||
("auto", DockerSupervisorNetworkMode::Auto),
|
||||
("host", DockerSupervisorNetworkMode::Host),
|
||||
("bridge", DockerSupervisorNetworkMode::Bridge),
|
||||
] {
|
||||
let config: DockerComputeConfig =
|
||||
toml::from_str(&format!("supervisor_network_mode = '{value}'")).unwrap();
|
||||
assert_eq!(config.supervisor_network_mode, expected);
|
||||
}
|
||||
assert!(toml::from_str::<DockerComputeConfig>("supervisor_network_mode = 'none'").is_err());
|
||||
assert!(
|
||||
serde_json::from_value::<DockerSandboxDriverConfig>(serde_json::json!({
|
||||
"supervisor_network_mode": "host"
|
||||
}))
|
||||
.is_err(),
|
||||
"workloads cannot select supervisor networking"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_container_create_body_limits_writable_runtime_storage_to_supervisor_ca() {
|
||||
let create_body = build_container_create_body(&test_sandbox(), &runtime_config()).unwrap();
|
||||
|
||||
@@ -121,6 +121,10 @@ struct Args {
|
||||
#[arg(long)]
|
||||
backend_descriptor_file: Option<PathBuf>,
|
||||
|
||||
/// Resolve the Docker host alias from this supervisor's driver-owned hosts file.
|
||||
#[arg(long)]
|
||||
host_gateway_from_hosts: bool,
|
||||
|
||||
/// Protected gateway-issued credentials for this exact sandbox launch.
|
||||
#[arg(long)]
|
||||
auth_bundle_file: Option<PathBuf>,
|
||||
@@ -177,14 +181,68 @@ fn backend_descriptor(args: &Args) -> Result<BackendDescriptor> {
|
||||
let path = args.backend_descriptor_file.as_deref().ok_or_else(|| {
|
||||
miette::miette!("--backend-descriptor-file is required for --role=isolation-backend")
|
||||
})?;
|
||||
let payload = std::fs::read(path)
|
||||
let mut payload = std::fs::read(path)
|
||||
.map_err(|error| miette::miette!("read backend descriptor {}: {error}", path.display()))?;
|
||||
if args.host_gateway_from_hosts {
|
||||
let hosts = std::fs::read_to_string("/etc/hosts")
|
||||
.map_err(|error| miette::miette!("read supervisor Docker host aliases: {error}"))?;
|
||||
let ip = docker_host_gateway_from_hosts(&hosts)?;
|
||||
let mut descriptor: openshell_sandbox_backend::boundary_protocol::SandboxRuntimeDescriptor =
|
||||
serde_json::from_slice(&payload).into_diagnostic()?;
|
||||
// Resolve before any workload is admitted, from the supervisor's mount
|
||||
// namespace. Workload /etc/hosts and DNS never supply this trusted IP.
|
||||
descriptor.host_gateway_ip = Some(ip);
|
||||
payload = serde_json::to_vec(&descriptor).into_diagnostic()?;
|
||||
}
|
||||
Ok(BackendDescriptor {
|
||||
backend_name: openshell_sandbox_backend::BACKEND_NAME.to_string(),
|
||||
payload,
|
||||
})
|
||||
}
|
||||
|
||||
fn docker_host_gateway_from_hosts(hosts: &str) -> Result<std::net::IpAddr> {
|
||||
let mut ips = std::collections::BTreeSet::new();
|
||||
for line in hosts.lines() {
|
||||
let mut fields = line.split('#').next().unwrap_or("").split_whitespace();
|
||||
let Some(ip) = fields
|
||||
.next()
|
||||
.and_then(|value| value.parse::<std::net::IpAddr>().ok())
|
||||
else {
|
||||
continue;
|
||||
};
|
||||
if fields.any(|host| host.eq_ignore_ascii_case("host.openshell.internal")) {
|
||||
ips.insert(ip);
|
||||
}
|
||||
}
|
||||
// Docker may expand host-gateway to both families. Prefer IPv4 consistently.
|
||||
let ipv4 = ips
|
||||
.iter()
|
||||
.copied()
|
||||
.filter(std::net::IpAddr::is_ipv4)
|
||||
.collect::<Vec<_>>();
|
||||
let candidates = if ipv4.is_empty() {
|
||||
ips.into_iter().collect()
|
||||
} else {
|
||||
ipv4
|
||||
};
|
||||
let [ip] = candidates.as_slice() else {
|
||||
return Err(miette::miette!(
|
||||
"expected one Docker host-gateway address per family in supervisor /etc/hosts"
|
||||
));
|
||||
};
|
||||
if ip.is_loopback()
|
||||
|| ip.is_unspecified()
|
||||
|| ip.is_multicast()
|
||||
|| matches!(ip, std::net::IpAddr::V4(ip) if ip.is_link_local() || ip.is_broadcast())
|
||||
|| matches!(ip, std::net::IpAddr::V6(ip) if ip.is_unicast_link_local() || ip.to_ipv4_mapped().is_some())
|
||||
{
|
||||
return Err(miette::miette!(
|
||||
"unsafe Docker host-gateway address in supervisor /etc/hosts"
|
||||
));
|
||||
}
|
||||
Ok(*ip)
|
||||
}
|
||||
|
||||
fn auth_bundle(args: &Args) -> Result<openshell_core::jwt::SupervisorAuthBundle> {
|
||||
let path = args.auth_bundle_file.as_deref().ok_or_else(|| {
|
||||
miette::miette!("--auth-bundle-file is required for --role=isolation-backend")
|
||||
@@ -229,6 +287,7 @@ fn validate_role_arguments(args: &Args) -> Result<()> {
|
||||
}
|
||||
SupervisorRole::NetworkProxy => {
|
||||
if args.backend_descriptor_file.is_some()
|
||||
|| args.host_gateway_from_hosts
|
||||
|| args.auth_bundle_file.is_some()
|
||||
|| args.sandbox_id.is_some()
|
||||
|| args.sandbox.is_some()
|
||||
@@ -565,6 +624,33 @@ mod tests {
|
||||
assert_eq!(directives("info,h2=debug"), "openshell=info,info");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn docker_host_gateway_uses_supervisor_alias_and_prefers_ipv4() {
|
||||
let hosts = "127.0.0.1 localhost\n192.168.65.254 host.openshell.internal host.docker.internal\nfd00::1 host.openshell.internal\n192.168.65.254 host.openshell.internal # duplicate\n";
|
||||
assert_eq!(
|
||||
docker_host_gateway_from_hosts(hosts).unwrap(),
|
||||
"192.168.65.254".parse::<std::net::IpAddr>().unwrap()
|
||||
);
|
||||
assert_eq!(
|
||||
docker_host_gateway_from_hosts("fd00::1 host.openshell.internal").unwrap(),
|
||||
"fd00::1".parse::<std::net::IpAddr>().unwrap()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn docker_host_gateway_rejects_missing_ambiguous_and_unsafe_aliases() {
|
||||
for hosts in [
|
||||
"192.168.65.254 host.docker.internal",
|
||||
"192.168.65.254 host.openshell.internal\n172.17.0.1 host.openshell.internal",
|
||||
"127.0.0.1 host.openshell.internal",
|
||||
"0.0.0.0 host.openshell.internal",
|
||||
"169.254.169.254 host.openshell.internal",
|
||||
"::ffff:127.0.0.1 host.openshell.internal",
|
||||
] {
|
||||
assert!(docker_host_gateway_from_hosts(hosts).is_err(), "{hosts}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn isolation_backend_is_the_default_role() {
|
||||
let directory = tempfile::tempdir().expect("temporary runtime descriptor directory");
|
||||
|
||||
@@ -123,7 +123,7 @@ The gateway can manage sandboxes through several runtimes.
|
||||
|
||||
| Runtime | Status | Notes |
|
||||
|---|---|---|
|
||||
| Docker | Supported for local development and single-machine gateways. | Requires Docker Desktop or Docker Engine on the gateway host. |
|
||||
| Docker | Supported for local development and single-machine gateways. | Requires an accessible Docker API socket. Supervisor networking defaults to host, or bridge for detected Sysbox/ECI workloads. See [Docker runtime setup](/how-it-works/sandboxes/runtimes#docker-driver) for endpoint and isolation requirements. |
|
||||
| Podman | Supported for rootless local and workstation workflows. | Requires a Podman-compatible socket and rootless networking setup. |
|
||||
| Kubernetes | Supported through the [OpenShell Helm chart](https://github.com/NVIDIA/OpenShell/blob/main/deploy/helm/openshell/README.md). | Requires a Kubernetes cluster supplied by the operator. |
|
||||
| MicroVM | Supported for VM-backed sandboxes. | Uses the VM compute driver and libkrun-based runtime. |
|
||||
|
||||
@@ -965,9 +965,11 @@ default_image = "nvcr.io/nvidia/base/ubuntu:24.04"
|
||||
image_pull_policy = "if_not_present"
|
||||
# Value assigned to the openshell.sandbox_namespace label on sandbox containers.
|
||||
sandbox_label = "docker-dev"
|
||||
# Optional override. When omitted, the host-networked supervisor uses the
|
||||
# gateway's primary loopback endpoint.
|
||||
grpc_endpoint = "https://127.0.0.1:17670"
|
||||
# auto uses bridge for Sysbox/ECI workloads, otherwise host networking.
|
||||
# Operator-owned; sandbox driver config cannot override this choice.
|
||||
supervisor_network_mode = "auto"
|
||||
# Optional override. Omit for automatic host or bridge callback selection.
|
||||
# grpc_endpoint = "https://gateway.example.com:17670"
|
||||
# Workload-side runtime. Defaults to the gateway version.
|
||||
# sandbox_runtime_image = "ghcr.io/nvidia/openshell/sandbox:<version>"
|
||||
# Supervisor runtime. Defaults to the gateway version.
|
||||
@@ -1003,6 +1005,35 @@ provider_spiffe_workload_api_socket = "/run/spire/agent.sock"
|
||||
Use `sandbox_label` for Docker configurations. The legacy
|
||||
`sandbox_namespace` key is rejected.
|
||||
|
||||
`supervisor_network_mode` accepts `auto` (the default), `host`, or `bridge`.
|
||||
On each supervisor launch, including restart, `auto` inspects the existing
|
||||
workload container's runtime. `sysbox-runc`, used by Docker Desktop Enhanced
|
||||
Container Isolation (ECI), selects bridge networking; other runtimes select
|
||||
host networking. This heuristic creates no diagnostic container. An explicit
|
||||
mode overrides it. The workload always uses `network=none`.
|
||||
|
||||
When `grpc_endpoint` is omitted, host mode uses the gateway's primary loopback
|
||||
endpoint. Bridge mode uses `host.docker.internal` and the gateway's bind port
|
||||
on Docker Desktop or for a wildcard listener. On native Linux, a concrete
|
||||
non-loopback listener uses its bind address. A Linux gateway bound only to
|
||||
loopback requires a bridge-reachable listener and endpoint; OpenShell does not
|
||||
automatically widen the listener. A gateway container with a remapped published
|
||||
port requires an explicit endpoint using that host port. Explicit endpoints
|
||||
retain their scheme, host, port, and TLS verification; bridge mode rejects
|
||||
loopback and unspecified endpoint addresses. HTTPS certificates must cover
|
||||
the selected endpoint hostname or IP.
|
||||
|
||||
In bridge mode, Docker resolves `host-gateway` for the supervisor's
|
||||
`host.openshell.internal` and `host.docker.internal` aliases independently of
|
||||
the gateway endpoint. The supervisor pins that driver-owned address before
|
||||
admitting the workload. On native Linux, host services listening only on
|
||||
loopback are unreachable through the bridge; expose them on a bridge-reachable
|
||||
interface or use host mode when permitted. Host mode on Docker Desktop requires
|
||||
host networking enabled and ECI disabled. Bridge selection removes that
|
||||
networking conflict but does not establish full ECI compatibility: the runtime
|
||||
must still pass OpenShell's Landlock and seccomp qualification. A containerized
|
||||
gateway under ECI also requires an administrator-approved Docker socket exception.
|
||||
|
||||
Docker accepts `http://` and `https://` proxy URLs in explicit
|
||||
`scheme://host:port` form. `no_proxy` bypasses only the corporate proxy;
|
||||
OpenShell policy still applies. Proxy URLs cannot embed credentials. Supply a
|
||||
|
||||
@@ -26,7 +26,7 @@ A gateway provisions sandboxes through the compute driver configured for that ga
|
||||
|
||||
| Runtime | Where sandboxes run | Best for |
|
||||
|---|---|---|
|
||||
| Docker | Containers on the gateway host. | Solo development, quick iteration, and single-machine gateways. |
|
||||
| Docker | Containers on the Docker daemon host. | Solo development, quick iteration, and single-machine gateways. |
|
||||
| Podman | Rootless containers on the gateway host. | Workstations that avoid a rootful Docker daemon. |
|
||||
| Kubernetes | Pods in an operator-managed cluster. | Shared clusters and cloud environments. |
|
||||
| MicroVM | VM-backed sandboxes. | Workflows that need VM-backed isolation. |
|
||||
|
||||
@@ -102,7 +102,7 @@ To pin specific GPUs, pass `cdi_devices` for Docker or Podman, or `gpu_device_id
|
||||
|
||||
## Docker Driver
|
||||
|
||||
[Docker](https://www.docker.com/get-started/) runs sandboxes as containers on the gateway host. Docker is also required to build sandbox images from local directories or Dockerfiles.
|
||||
[Docker](https://www.docker.com/get-started/) runs sandboxes as containers on the Docker daemon host. The gateway accesses the daemon through its configured Unix socket. Docker is also required to build sandbox images from local directories or Dockerfiles.
|
||||
|
||||
```toml
|
||||
[openshell.gateway]
|
||||
@@ -112,9 +112,13 @@ compute_driver = "docker"
|
||||
socket_path = "/var/run/docker.sock"
|
||||
```
|
||||
|
||||
Common options in `[openshell.drivers.docker]` are `socket_path`, `grpc_endpoint`, `sandbox_runtime_image`, `supervisor_image`, `image_pull_policy`, and `sandbox_pids_limit`. When `socket_path` is unset, the driver uses the socket found by auto-detection.
|
||||
Common options in `[openshell.drivers.docker]` are `socket_path`, `grpc_endpoint`, `supervisor_network_mode`, `sandbox_runtime_image`, `supervisor_image`, `image_pull_policy`, and `sandbox_pids_limit`. When `socket_path` is unset, the driver uses the socket found by auto-detection.
|
||||
|
||||
Docker Desktop must have host networking enabled, and it cannot use Enhanced Container Isolation. Set `grpc_endpoint` when sandboxes cannot reach the gateway on host loopback. For GPU sandboxes, configure Docker CDI before starting the gateway.
|
||||
`supervisor_network_mode` defaults to `auto`: the driver uses bridge networking for workloads whose inspected runtime is `sysbox-runc` (Docker Desktop Enhanced Container Isolation), and host networking otherwise. Set `host` or `bridge` to override the heuristic. The workload always retains `network=none`.
|
||||
|
||||
Host mode on Docker Desktop requires host networking enabled and Enhanced Container Isolation disabled. Bridge mode automatically uses `host.docker.internal` for a gateway on the Desktop host. On native Linux, the gateway and host services must listen on bridge-reachable addresses. Set `grpc_endpoint` for remote gateways, remapped published ports, or other custom layouts. Endpoint selection preserves TLS verification and does not change the gateway's listener. See [Docker configuration](/how-it-works/gateways/configuration#docker).
|
||||
|
||||
Bridge selection does not establish full Enhanced Container Isolation compatibility; the sandbox must still pass its Landlock and seccomp qualification. A gateway running in an ECI container requires an administrator-approved Docker socket exception. For GPU sandboxes, configure Docker CDI before starting the gateway.
|
||||
|
||||
### Docker Corporate Proxy Egress
|
||||
|
||||
|
||||
@@ -84,6 +84,11 @@ name = "docker_corporate_proxy"
|
||||
path = "tests/docker_corporate_proxy.rs"
|
||||
required-features = ["e2e-docker"]
|
||||
|
||||
[[test]]
|
||||
name = "docker_supervisor_networking"
|
||||
path = "tests/docker_supervisor_networking.rs"
|
||||
required-features = ["e2e-docker"]
|
||||
|
||||
[[test]]
|
||||
name = "driver_config_volume"
|
||||
path = "tests/driver_config_volume.rs"
|
||||
|
||||
@@ -0,0 +1,127 @@
|
||||
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
#![cfg(feature = "e2e-docker")]
|
||||
|
||||
//! Run with both the default auto mode and forced bridge mode to exercise
|
||||
//! authenticated gateway callbacks, interactive access, and supervisor restart.
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
use openshell_e2e::harness::cli::{run_cli, wait_for_sandbox_phase};
|
||||
use openshell_e2e::harness::sandbox::SandboxGuard;
|
||||
|
||||
async fn inspect_role(name: &str, role: &str) -> serde_json::Value {
|
||||
let listed = tokio::process::Command::new("docker")
|
||||
.args([
|
||||
"ps",
|
||||
"-aq",
|
||||
"--filter",
|
||||
&format!("label=openshell.ai/sandbox-name={name}"),
|
||||
"--filter",
|
||||
&format!("label=openshell.ai/isolation-role={role}"),
|
||||
])
|
||||
.output()
|
||||
.await
|
||||
.expect("list sandbox containers");
|
||||
assert!(listed.status.success());
|
||||
let id = String::from_utf8(listed.stdout).unwrap();
|
||||
let id = id.trim();
|
||||
assert!(!id.is_empty(), "missing {role} container for {name}");
|
||||
let inspected = tokio::process::Command::new("docker")
|
||||
.args(["inspect", id])
|
||||
.output()
|
||||
.await
|
||||
.expect("inspect sandbox container");
|
||||
assert!(inspected.status.success());
|
||||
serde_json::from_slice::<Vec<serde_json::Value>>(&inspected.stdout)
|
||||
.unwrap()
|
||||
.remove(0)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn supervisor_networking_preserves_boundary_and_survives_restart() {
|
||||
let policy = tempfile::NamedTempFile::new().expect("create policy file");
|
||||
std::fs::write(
|
||||
policy.path(),
|
||||
r"version: 1
|
||||
filesystem_policy:
|
||||
include_workdir: true
|
||||
read_only: [/usr, /bin, /lib, /proc, /dev/urandom, /app, /etc, /var/log]
|
||||
read_write: [/sandbox, /tmp, /dev/null]
|
||||
landlock:
|
||||
compatibility: best_effort
|
||||
network_policies:
|
||||
host_gateway:
|
||||
name: host_gateway
|
||||
endpoints:
|
||||
- {host: host.openshell.internal, port: 8080, protocol: tcp}
|
||||
binaries:
|
||||
- {path: /usr/bin/getent}
|
||||
",
|
||||
)
|
||||
.expect("write policy file");
|
||||
let mut sandbox = SandboxGuard::create(&["--policy", policy.path().to_str().unwrap()])
|
||||
.await
|
||||
.expect("create sandbox");
|
||||
let workload = inspect_role(&sandbox.name, "sandbox").await;
|
||||
let requested = std::env::var("OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE")
|
||||
.unwrap_or_else(|_| "auto".into());
|
||||
let expected = if requested == "bridge"
|
||||
|| (requested == "auto" && workload["HostConfig"]["Runtime"] == "sysbox-runc")
|
||||
{
|
||||
"bridge"
|
||||
} else {
|
||||
"host"
|
||||
};
|
||||
assert_eq!(workload["HostConfig"]["NetworkMode"], "none");
|
||||
assert!(
|
||||
workload["NetworkSettings"]["Networks"]
|
||||
.as_object()
|
||||
.is_none_or(|networks| networks.keys().all(|name| name == "none"))
|
||||
);
|
||||
let supervisor = inspect_role(&sandbox.name, "supervisor").await;
|
||||
assert_eq!(supervisor["HostConfig"]["NetworkMode"], expected);
|
||||
assert!(
|
||||
sandbox
|
||||
.exec(&["sh", "-c", "echo networking-ready"])
|
||||
.await
|
||||
.unwrap()
|
||||
.contains("networking-ready")
|
||||
);
|
||||
assert!(
|
||||
!sandbox
|
||||
.exec(&["/usr/bin/getent", "hosts", "host.openshell.internal"])
|
||||
.await
|
||||
.expect("resolve policy-approved host alias")
|
||||
.trim()
|
||||
.is_empty()
|
||||
);
|
||||
let (output, code) = run_cli(&["sandbox", "stop", &sandbox.name]).await;
|
||||
assert_eq!(code, 0, "{output}");
|
||||
wait_for_sandbox_phase(&sandbox.name, "Stopped", Duration::from_secs(60))
|
||||
.await
|
||||
.unwrap();
|
||||
let (output, code) = run_cli(&["sandbox", "start", &sandbox.name]).await;
|
||||
assert_eq!(code, 0, "{output}");
|
||||
assert!(
|
||||
sandbox
|
||||
.exec(&["sh", "-c", "echo networking-restarted"])
|
||||
.await
|
||||
.unwrap()
|
||||
.contains("networking-restarted")
|
||||
);
|
||||
assert!(
|
||||
!sandbox
|
||||
.exec(&["/usr/bin/getent", "hosts", "host.openshell.internal"])
|
||||
.await
|
||||
.expect("resolve host alias after restart")
|
||||
.trim()
|
||||
.is_empty()
|
||||
);
|
||||
assert_eq!(
|
||||
inspect_role(&sandbox.name, "supervisor").await["HostConfig"]["NetworkMode"],
|
||||
expected
|
||||
);
|
||||
sandbox.cleanup().await;
|
||||
}
|
||||
@@ -22,6 +22,8 @@
|
||||
# SUPERVISOR_IMAGE=... (common test-wrapper override)
|
||||
# OPENSHELL_SUPERVISOR_IMAGE=... (existing compatibility override)
|
||||
# OPENSHELL_DOCKER_SUPERVISOR_IMAGE=... (Docker-specific override)
|
||||
# OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE=auto|host|bridge
|
||||
# OPENSHELL_E2E_DOCKER_AUTOMATIC_ENDPOINT=1 omits the callback override.
|
||||
#
|
||||
# The default sandbox image uses a mutable tag. This wrapper refreshes it
|
||||
# before starting the gateway, while the Docker driver defaults to
|
||||
@@ -570,6 +572,11 @@ if connect_current_container_to_docker_network "${DOCKER_NETWORK_NAME}"; then
|
||||
SUPERVISOR_GATEWAY_HOST="${GATEWAY_HOST_ALIAS_IP}"
|
||||
else
|
||||
GATEWAY_HOST_ALIAS_IP=""
|
||||
if [ "${OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE:-auto}" = "bridge" ]; then
|
||||
# This ephemeral test gateway deliberately admits bridge-side connections.
|
||||
GATEWAY_BIND_IP="0.0.0.0"
|
||||
SUPERVISOR_GATEWAY_HOST="host.docker.internal"
|
||||
fi
|
||||
fi
|
||||
|
||||
PKI_DIR="${WORKDIR}/pki"
|
||||
@@ -619,7 +626,10 @@ GATEWAY_CONFIG="${STATE_DIR}/gateway.toml"
|
||||
else
|
||||
printf 'allow_driver_config = true\n'
|
||||
printf 'sandbox_label = %s\n' "$(toml_string "${E2E_NAMESPACE}")"
|
||||
printf 'grpc_endpoint = %s\n' "$(toml_string "${GATEWAY_ENDPOINT}")"
|
||||
if [ "${OPENSHELL_E2E_DOCKER_AUTOMATIC_ENDPOINT:-0}" != "1" ]; then
|
||||
printf 'grpc_endpoint = %s\n' "$(toml_string "${GATEWAY_ENDPOINT}")"
|
||||
fi
|
||||
printf 'supervisor_network_mode = %s\n' "$(toml_string "${OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE:-auto}")"
|
||||
printf 'default_image = %s\n' "$(toml_string "${SANDBOX_IMAGE}")"
|
||||
printf 'image_pull_policy = %s\n' "$(toml_string "${SANDBOX_IMAGE_PULL_POLICY}")"
|
||||
printf 'enable_bind_mounts = true\n'
|
||||
@@ -633,7 +643,10 @@ GATEWAY_CONFIG="${STATE_DIR}/gateway.toml"
|
||||
if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then
|
||||
{
|
||||
printf 'sandbox_label = %s\n' "$(toml_string "${E2E_NAMESPACE}")"
|
||||
printf 'grpc_endpoint = %s\n' "$(toml_string "${GATEWAY_ENDPOINT}")"
|
||||
if [ "${OPENSHELL_E2E_DOCKER_AUTOMATIC_ENDPOINT:-0}" != "1" ]; then
|
||||
printf 'grpc_endpoint = %s\n' "$(toml_string "${GATEWAY_ENDPOINT}")"
|
||||
fi
|
||||
printf 'supervisor_network_mode = %s\n' "$(toml_string "${OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE:-auto}")"
|
||||
printf 'default_image = %s\n' "$(toml_string "${SANDBOX_IMAGE}")"
|
||||
printf 'image_pull_policy = %s\n' "$(toml_string "${SANDBOX_IMAGE_PULL_POLICY}")"
|
||||
printf 'guest_tls_ca = %s\n' "$(toml_string "${PKI_DIR}/ca.crt")"
|
||||
|
||||
@@ -309,9 +309,19 @@ Common findings:
|
||||
- SSH host-key startup errors: use matching gateway, compute-driver, and supervisor releases. The gateway retains each sandbox's SSH key in its configured credential store and sends it only through the supervisor bootstrap bundle. Check credential-driver availability and compare the public `host_key_fingerprint` from sandbox JSON output; never print the bootstrap bundle or private key. A missing stored key for a sandbox with a fingerprint is an error, not permission to replace its identity. See the [sandbox SSH identity documentation](https://docs.nvidia.com/openshell/latest/how-it-works/sandboxes/overview).
|
||||
- Calls to an external tool server fail while the sandbox is Ready: inspect `Tool server connections` in `openshell sandbox get <name>`. For configured MCP-over-HTTP endpoints, JSON output exposes each address together with `last_result` and `last_reported_at` in `endpoint_statuses`. Select the endpoint by host, path, and ports, then check the reported failure boundary. `last_reported_at` records gateway acceptance time and can advance when retained evidence is accepted after a reset. Results do not expire or prove current availability; `HttpResponseReceived` can still contain a tool error. If several paths share a host and port, a failure before the path is known remains in logs. Verify the actual operation when current tool availability matters.
|
||||
- On Docker Desktop, repeated `Policy fetch failed after 5 attempts` messages
|
||||
can mean host networking is disabled. Enable host networking in Docker
|
||||
Desktop, ensure Enhanced Container Isolation is disabled, and verify the
|
||||
gateway's primary endpoint is reachable from a host-networked container.
|
||||
can mean the supervisor's selected network mode cannot reach the gateway.
|
||||
Check the gateway's `Selected Docker supervisor networking` log. The default
|
||||
`supervisor_network_mode = "auto"` selects bridge for inspected `sysbox-runc`
|
||||
workloads (ECI), and host otherwise. Host mode requires Desktop host
|
||||
networking enabled and ECI disabled; set `bridge` explicitly when neither
|
||||
feature is enabled. Bridge mode uses `host.docker.internal` for the Desktop
|
||||
host and requires a bridge-reachable listener on native Linux. Explicit
|
||||
`grpc_endpoint` values must be reachable in the selected mode, including any
|
||||
remapped published port, with matching TLS certificate names. Mode selection
|
||||
does not bypass Landlock/seccomp qualification. A gateway container under ECI
|
||||
also needs an administrator-approved Docker socket exception. See the
|
||||
published [Docker configuration](https://docs.nvidia.com/openshell/latest/how-it-works/gateways/configuration.md)
|
||||
for the current networking options.
|
||||
- Sandbox runtime image exits before printing `openshell-sandbox --version`: verify the configured image contains a static executable at `/openshell-sandbox`.
|
||||
- A sandbox with explicit `protocol: tcp` endpoints fails before workload readiness: confirm the selected isolation backend advertises TCP mediation, then inspect the sandbox and supervisor logs for protected-channel setup or listener failures. A driver that cannot supply the required outer egress fence and authenticated runtime channel must reject the policy before starting the agent.
|
||||
- Supervisor runtime validation fails: verify `supervisor_image` contains an `/openshell-supervisor` executable from the same release as the sandbox runtime, and that the dynamic loader and shared libraries it links against are available inside that image. `docker run --rm --network none --entrypoint /openshell-supervisor <supervisor_image> --version` should print that release; a `no such file or directory` error for a binary that exists means the loader or a library is missing. The supervisor runs from its own image and does not need to be static; only `/openshell-sandbox` must be.
|
||||
|
||||
@@ -301,6 +301,14 @@ description = "Run Docker conformance and Rust e2e tests against a standalone ga
|
||||
depends = ["e2e:conformance:build"]
|
||||
run = "OPENSHELL_CONFORMANCE_BIN=\"${OPENSHELL_CONFORMANCE_BIN:-$PWD/target/debug/openshell-conformance}\" e2e/rust/e2e-docker.sh"
|
||||
|
||||
["e2e:docker:supervisor-networking"]
|
||||
description = "Exercise Docker supervisor networking and restart in auto and bridge modes"
|
||||
depends = ["e2e:workload:build"]
|
||||
run = [
|
||||
"e2e/with-docker-gateway.sh cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-docker --test docker_supervisor_networking",
|
||||
"OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE=bridge OPENSHELL_E2E_DOCKER_AUTOMATIC_ENDPOINT=1 e2e/with-docker-gateway.sh cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-docker --test docker_supervisor_networking",
|
||||
]
|
||||
|
||||
["e2e:mechanistic-existing-endpoint"]
|
||||
description = "Run #2821 existing inspected-endpoint auto-approval regression"
|
||||
run = [
|
||||
|
||||
Reference in New Issue
Block a user