mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-02 07:34:45 +08:00
fix(macos): restore Homebrew sandbox callbacks (#2739)
* fix(macos): restore Docker gateway callbacks Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(gateway): reuse reachable primary callback listener Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com>
This commit is contained in:
@@ -178,6 +178,13 @@ Common findings:
|
||||
- A workdir rejected as a special filesystem or OpenShell control-path collision cannot be made valid with permissions. Move the image workdir away from kernel-backed mounts and the concrete supervisor, TLS, token, runtime, and socket paths named in the error.
|
||||
- Docker driver cannot initialize because it cannot find `openshell-sandbox`: verify `OPENSHELL_DOCKER_SUPERVISOR_BIN`, the sibling binary next to `openshell-gateway`, or the configured supervisor image contains `/openshell-sandbox`.
|
||||
- Sandbox never registers: check gateway logs and supervisor callback endpoint.
|
||||
- On macOS, repeated `Policy fetch failed after 5 attempts` messages with a
|
||||
Homebrew gateway bound to `[::1]:17670` indicate that the Docker
|
||||
`host-gateway` IPv4 route has no matching callback listener. Current releases
|
||||
leave `bind_address` unset in the Homebrew config, use the built-in
|
||||
`127.0.0.1:17670` primary listener, and reuse it for authenticated sandbox
|
||||
callbacks. On an older release, set `bind_address = "127.0.0.1:17670"` or
|
||||
upgrade.
|
||||
- Supervisor image exits before printing `openshell-sandbox --version`: the image should be the scratch supervisor image from `deploy/docker/Dockerfile.supervisor` and must contain a static executable at `/openshell-sandbox`.
|
||||
- `mise run e2e:docker:gpu` fails with `docker info --format json did not report any discovered NVIDIA CDI GPU devices`: Docker may report `CDISpecDirs` while still having no generated NVIDIA CDI specs. Verify `.DiscoveredDevices` contains entries such as `nvidia.com/gpu=all`, verify `/etc/cdi` or `/var/run/cdi` contains a generated NVIDIA spec, and check that `nvidia-cdi-refresh.service` and `nvidia-cdi-refresh.path` from NVIDIA Container Toolkit are enabled and healthy. The service is a one-shot unit, so `inactive (dead)` can be normal after a successful run; use `systemctl status` and `journalctl` to distinguish success from a skipped or failed refresh. NVIDIA recommends enabling the path and service units, and restarting `nvidia-cdi-refresh.service` to regenerate missing or stale CDI specs. If specs are generated but Docker still reports no discovered devices, restart Docker or reload the daemon and re-check `docker info`.
|
||||
|
||||
@@ -207,14 +214,9 @@ Common findings:
|
||||
error: inspect `podman info --debug`, the configured Podman network, and the
|
||||
host's IPv4 default route. Rootless pasta uses the private source address
|
||||
selected by that route; rootful Podman uses the bridge gateway address.
|
||||
- Callback discovery reports that the requested address equals the primary
|
||||
listener: configure a distinct primary address. For Podman Machine, bind the
|
||||
primary listener to IPv6 loopback, for example
|
||||
`bind_address = "[::1]:17670"`, and register the CLI endpoint as
|
||||
`https://localhost:17670`. The generated certificate includes `localhost`,
|
||||
while a raw `https://[::1]:17670` endpoint can fail TLS setup with
|
||||
`invalid dns name`. This leaves `127.0.0.1:17670` available for the
|
||||
callback-only listener.
|
||||
- Current gateways reuse the primary listener when it covers Podman's callback
|
||||
address. If the primary does not cover that address, inspect the gateway
|
||||
startup logs for the additional callback-only listener and its provenance.
|
||||
- Rootless slirp4netns, another named helper, or missing helper metadata
|
||||
requires an explicitly remote `grpc_endpoint`. An explicit `host_gateway_ip`
|
||||
cannot bypass slirp4netns host-loopback isolation. Do not work around
|
||||
@@ -544,7 +546,7 @@ openshell logs <sandbox-name>
|
||||
| `openshell status` fails | Gateway endpoint unreachable or auth mismatch | `openshell gateway info`, gateway logs |
|
||||
| Gateway starts but sandbox create fails | Compute driver cannot reach runtime | Docker/Podman/Kubernetes/VM driver logs |
|
||||
| Gateway exits while resolving compute-driver listener requirements | Callback alias topology is unsupported, the Podman network cannot be inspected, or the selected address is not private/authorized | Gateway startup error, `podman info --debug`, Podman network inspection, host IPv4 default route |
|
||||
| Admin, health, reflection, or HTTP request is denied on a Docker/Podman callback address | Negotiated callback listeners intentionally expose only sandbox-callable gRPC methods | Retry through the gateway's primary endpoint; inspect the listener-purpose startup log if the address was unexpected |
|
||||
| Admin, health, reflection, or HTTP request is denied on an additional Docker/Podman callback-only listener | Additional callback listeners intentionally expose only sandbox-callable gRPC methods | Retry through the gateway's primary endpoint; inspect the listener-purpose startup log if the address was unexpected |
|
||||
| Docker or Podman sandbox never registers | Wrong callback endpoint or supervisor startup failure | Gateway logs and sandbox container logs |
|
||||
| Docker GPU e2e fails before GPU sandbox comparison | NVIDIA CDI specs are missing or Docker has not discovered them | `docker info --format '{{json .DiscoveredDevices}}'`, `/etc/cdi`, `/var/run/cdi`, `nvidia-cdi-refresh.service` |
|
||||
| Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod <pod>` |
|
||||
|
||||
@@ -37,19 +37,17 @@ health, metrics, or tunnel routes. The plaintext service router also rejects
|
||||
browser requests whose Fetch Metadata, Origin, or Referer headers indicate a
|
||||
cross-origin or sibling-subdomain request.
|
||||
|
||||
Docker and Podman may negotiate additional listeners that make the gateway
|
||||
reachable from their local sandbox network topology. Those listeners accept
|
||||
Docker and Podman report the local address through which their sandboxes can
|
||||
reach the gateway. When the primary listener covers that address, the gateway
|
||||
reuses it; sandbox JWT authentication and its RPC allowlist remain the callback
|
||||
authorization boundary. When the primary listener does not cover the address,
|
||||
the gateway adds a callback-only listener. Additional callback listeners accept
|
||||
only gRPC methods classified as sandbox-callable by the gateway's generated
|
||||
authorization metadata. They reject user and administrator APIs, health,
|
||||
reflection, non-callback inference APIs, and HTTP routes before normal request
|
||||
authentication. The operator-configured primary listener retains the full
|
||||
multiplexed API surface.
|
||||
|
||||
The gateway rejects a callback requirement that resolves to the exact primary
|
||||
listener address because one socket cannot preserve two authorization scopes.
|
||||
A wildcard primary listener may cover a callback address because the accepted
|
||||
connection's concrete local address still selects the callback-only scope.
|
||||
|
||||
The `rpc_auth` classification is also the source of truth for negotiated
|
||||
listener exposure: marking an RPC as `sandbox` or `dual` makes it callable on
|
||||
these listeners. Review such changes as both authorization and network-surface
|
||||
@@ -699,8 +697,10 @@ system entry instead of pretending to delete package-manager owned state.
|
||||
- Compute runtimes own the mechanics of starting workloads and injecting
|
||||
callback configuration.
|
||||
- Docker-backed local gateways use Docker's `host-gateway` callback alias on
|
||||
macOS and Docker Desktop-style runtimes. Native Linux Docker may expose an
|
||||
additional bridge-gateway listener because the host can bind that bridge IP.
|
||||
macOS and Docker Desktop-style runtimes. They request IPv4 loopback callback
|
||||
reachability and add a listener only when the primary does not cover it.
|
||||
Native Linux Docker may expose an additional bridge-gateway listener because
|
||||
the host can bind that bridge IP.
|
||||
- Podman-backed macOS gateways use gvproxy's host-loopback IP for sandbox host
|
||||
aliases by default so stale Podman machine images do not need Podman's
|
||||
`host-gateway` resolver. Linux Podman keeps the resolver unless
|
||||
|
||||
@@ -22,6 +22,7 @@ use tokio::sync::Mutex;
|
||||
use tonic::service::interceptor::InterceptedService;
|
||||
use tonic::transport::{Certificate, Channel, ClientTlsConfig, Endpoint, Identity};
|
||||
use tracing::debug;
|
||||
use url::{Host, Url};
|
||||
|
||||
/// Concrete gRPC client type used by all commands.
|
||||
pub type GrpcClient = OpenShellClient<InterceptedService<Channel, EdgeAuthInterceptor>>;
|
||||
@@ -223,12 +224,33 @@ pub fn build_rustls_config(materials: &TlsMaterials) -> Result<rustls::ClientCon
|
||||
.into_diagnostic()
|
||||
}
|
||||
|
||||
pub fn build_tonic_tls_config(materials: &TlsMaterials) -> ClientTlsConfig {
|
||||
pub fn build_tonic_tls_config(server: &str, materials: &TlsMaterials) -> Result<ClientTlsConfig> {
|
||||
let ca_cert = Certificate::from_pem(materials.ca.clone());
|
||||
let identity = Identity::from_pem(materials.cert.clone(), materials.key.clone());
|
||||
ClientTlsConfig::new()
|
||||
.ca_certificate(ca_cert)
|
||||
.identity(identity)
|
||||
tls_config_with_server_name(
|
||||
server,
|
||||
ClientTlsConfig::new()
|
||||
.ca_certificate(ca_cert)
|
||||
.identity(identity),
|
||||
)
|
||||
}
|
||||
|
||||
fn tls_config_with_server_name(server: &str, config: ClientTlsConfig) -> Result<ClientTlsConfig> {
|
||||
Ok(config.domain_name(tls_server_name(server)?))
|
||||
}
|
||||
|
||||
fn tls_server_name(server: &str) -> Result<String> {
|
||||
let endpoint = Url::parse(server)
|
||||
.into_diagnostic()
|
||||
.wrap_err_with(|| format!("invalid gateway endpoint '{server}'"))?;
|
||||
let host = endpoint
|
||||
.host()
|
||||
.ok_or_else(|| miette::miette!("gateway endpoint '{server}' has no host"))?;
|
||||
Ok(match host {
|
||||
Host::Domain(domain) => domain.to_string(),
|
||||
Host::Ipv4(address) => address.to_string(),
|
||||
Host::Ipv6(address) => address.to_string(),
|
||||
})
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
@@ -402,22 +424,20 @@ pub async fn build_channel(server: &str, tls: &TlsOptions) -> Result<Channel> {
|
||||
// Bearer auth over HTTPS: use mTLS certs for the transport layer when
|
||||
// available (server may still require client certs), and layer the
|
||||
// Bearer token on top via the interceptor.
|
||||
require_tls_materials(server, tls).map_or_else(
|
||||
|_| {
|
||||
let resolved = tls.with_default_paths(server);
|
||||
resolved
|
||||
.ca
|
||||
.as_ref()
|
||||
.and_then(|ca_path| std::fs::read(ca_path).ok())
|
||||
.map_or_else(
|
||||
|| ClientTlsConfig::new().with_enabled_roots(),
|
||||
|ca_pem| {
|
||||
ClientTlsConfig::new().ca_certificate(Certificate::from_pem(ca_pem))
|
||||
},
|
||||
)
|
||||
},
|
||||
|materials| build_tonic_tls_config(&materials),
|
||||
)
|
||||
if let Ok(materials) = require_tls_materials(server, tls) {
|
||||
build_tonic_tls_config(server, &materials)?
|
||||
} else {
|
||||
let resolved = tls.with_default_paths(server);
|
||||
let config = resolved
|
||||
.ca
|
||||
.as_ref()
|
||||
.and_then(|ca_path| std::fs::read(ca_path).ok())
|
||||
.map_or_else(
|
||||
|| ClientTlsConfig::new().with_enabled_roots(),
|
||||
|ca_pem| ClientTlsConfig::new().ca_certificate(Certificate::from_pem(ca_pem)),
|
||||
);
|
||||
tls_config_with_server_name(server, config)?
|
||||
}
|
||||
} else if tls.edge_token.is_some() {
|
||||
// Edge bearer mode — routed through tunnel above; if we reach here
|
||||
// the server is not HTTPS so connect plaintext.
|
||||
@@ -425,7 +445,7 @@ pub async fn build_channel(server: &str, tls: &TlsOptions) -> Result<Channel> {
|
||||
} else {
|
||||
// Standard mTLS: private CA + client cert.
|
||||
let materials = require_tls_materials(server, tls)?;
|
||||
build_tonic_tls_config(&materials)
|
||||
build_tonic_tls_config(server, &materials)?
|
||||
};
|
||||
endpoint = endpoint.tls_config(tls_config).into_diagnostic()?;
|
||||
endpoint.connect().await.into_diagnostic()
|
||||
@@ -451,3 +471,25 @@ pub async fn grpc_inference_client(server: &str, tls: &TlsOptions) -> Result<Grp
|
||||
let interceptor = interceptor_from_tls(tls)?;
|
||||
Ok(InferenceClient::with_interceptor(channel, interceptor))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::tls_server_name;
|
||||
|
||||
#[test]
|
||||
fn tls_server_name_normalizes_bracketed_ipv6_endpoint() {
|
||||
assert_eq!(
|
||||
tls_server_name("https://[::1]:17670").unwrap(),
|
||||
"::1",
|
||||
"rustls accepts an unbracketed IPv6 address as an IP server name"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tls_server_name_preserves_dns_endpoint() {
|
||||
assert_eq!(
|
||||
tls_server_name("https://localhost:17670").unwrap(),
|
||||
"localhost"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -67,8 +67,10 @@ Docker containers join an OpenShell-managed bridge network. The driver injects
|
||||
`host.openshell.internal` and `host.docker.internal` so supervisors have stable
|
||||
names for reaching the gateway host. On Docker Desktop, Colima, Rancher
|
||||
Desktop, OrbStack, and macOS-hosted gateways, those names use Docker's
|
||||
`host-gateway` alias. On native Linux Docker, the gateway also binds the bridge
|
||||
gateway IP so containers can call back to the host process.
|
||||
`host-gateway` alias. The driver requests a separate IPv4 loopback callback
|
||||
listener when the primary listener does not already cover it. On native Linux
|
||||
Docker, the gateway also binds the bridge gateway IP so containers can call
|
||||
back to the host process.
|
||||
|
||||
## Container Contract
|
||||
|
||||
|
||||
@@ -55,7 +55,7 @@ use openshell_core::proto_struct::{
|
||||
use openshell_core::{Config, Error, Result as CoreResult};
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::io::Read;
|
||||
use std::net::{IpAddr, SocketAddr};
|
||||
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::pin::Pin;
|
||||
use std::sync::Arc;
|
||||
@@ -177,6 +177,7 @@ struct DockerDriverRuntimeConfig {
|
||||
grpc_endpoint: String,
|
||||
network_name: String,
|
||||
gateway_route: DockerGatewayRoute,
|
||||
gateway_callback_bind_address: Option<SocketAddr>,
|
||||
ssh_socket_path: String,
|
||||
stop_timeout_secs: u32,
|
||||
log_level: String,
|
||||
@@ -428,6 +429,8 @@ impl DockerComputeDriver {
|
||||
let host_gateway_ip = parse_optional_host_gateway_ip(&docker_config.host_gateway_ip)?;
|
||||
let gateway_route =
|
||||
docker_gateway_route(&info, bridge_gateway_ip, gateway_port, host_gateway_ip);
|
||||
let gateway_callback_bind_address =
|
||||
docker_gateway_callback_bind_address(&gateway_route, config.bind_address);
|
||||
let mut docker_config = docker_config.clone();
|
||||
if docker_config.grpc_endpoint.trim().is_empty() {
|
||||
let scheme = if docker_guest_tls_configured(&docker_config) {
|
||||
@@ -456,6 +459,7 @@ impl DockerComputeDriver {
|
||||
grpc_endpoint,
|
||||
network_name,
|
||||
gateway_route,
|
||||
gateway_callback_bind_address,
|
||||
ssh_socket_path: docker_config.ssh_socket_path.clone(),
|
||||
stop_timeout_secs: DEFAULT_STOP_TIMEOUT_SECS,
|
||||
log_level: config.log_level.clone(),
|
||||
@@ -1577,15 +1581,19 @@ impl ComputeDriver for DockerComputeDriver {
|
||||
&self,
|
||||
_request: Request<GetGatewayListenerRequirementsRequest>,
|
||||
) -> Result<Response<GetGatewayListenerRequirementsResponse>, Status> {
|
||||
let requirements = match self.config.gateway_route {
|
||||
DockerGatewayRoute::Bridge { bind_address, .. } => {
|
||||
vec![GatewayListenerRequirement {
|
||||
reason: "docker managed bridge gateway".to_string(),
|
||||
selector: Some(Selector::ExactBindAddress(bind_address.to_string())),
|
||||
}]
|
||||
}
|
||||
DockerGatewayRoute::HostGateway => Vec::new(),
|
||||
};
|
||||
let requirements =
|
||||
self.config
|
||||
.gateway_callback_bind_address
|
||||
.map_or_else(Vec::new, |bind_address| {
|
||||
vec![GatewayListenerRequirement {
|
||||
reason: match self.config.gateway_route {
|
||||
DockerGatewayRoute::Bridge { .. } => "docker managed bridge gateway",
|
||||
DockerGatewayRoute::HostGateway => "docker host-gateway IPv4 loopback",
|
||||
}
|
||||
.to_string(),
|
||||
selector: Some(Selector::ExactBindAddress(bind_address.to_string())),
|
||||
}]
|
||||
});
|
||||
Ok(Response::new(GetGatewayListenerRequirementsResponse {
|
||||
requirements,
|
||||
}))
|
||||
@@ -2793,6 +2801,22 @@ fn docker_gateway_route_for_host(
|
||||
}
|
||||
}
|
||||
|
||||
fn docker_gateway_callback_bind_address(
|
||||
route: &DockerGatewayRoute,
|
||||
primary_bind_address: SocketAddr,
|
||||
) -> Option<SocketAddr> {
|
||||
match route {
|
||||
DockerGatewayRoute::Bridge { bind_address, .. } => Some(*bind_address),
|
||||
DockerGatewayRoute::HostGateway => match primary_bind_address.ip() {
|
||||
IpAddr::V4(ip) if ip.is_unspecified() || ip == Ipv4Addr::LOCALHOST => None,
|
||||
_ => Some(SocketAddr::new(
|
||||
IpAddr::V4(Ipv4Addr::LOCALHOST),
|
||||
primary_bind_address.port(),
|
||||
)),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
fn host_runtime_requires_host_gateway_alias() -> bool {
|
||||
cfg!(target_os = "macos")
|
||||
}
|
||||
|
||||
@@ -102,6 +102,10 @@ fn runtime_config() -> DockerDriverRuntimeConfig {
|
||||
),
|
||||
host_alias_ip: IpAddr::V4(Ipv4Addr::new(172, 18, 0, 1)),
|
||||
},
|
||||
gateway_callback_bind_address: Some(SocketAddr::new(
|
||||
IpAddr::V4(Ipv4Addr::new(172, 18, 0, 1)),
|
||||
DEFAULT_SERVER_PORT,
|
||||
)),
|
||||
ssh_socket_path: "/run/openshell/ssh.sock".to_string(),
|
||||
stop_timeout_secs: DEFAULT_STOP_TIMEOUT_SECS,
|
||||
log_level: "info".to_string(),
|
||||
@@ -186,6 +190,7 @@ async fn gateway_listener_requirements_report_managed_bridge_address() {
|
||||
async fn gateway_listener_requirements_are_empty_for_host_gateway_route() {
|
||||
let mut config = runtime_config();
|
||||
config.gateway_route = DockerGatewayRoute::HostGateway;
|
||||
config.gateway_callback_bind_address = None;
|
||||
let driver = test_driver_with_config(config);
|
||||
|
||||
let response = driver
|
||||
@@ -197,6 +202,26 @@ async fn gateway_listener_requirements_are_empty_for_host_gateway_route() {
|
||||
assert!(response.requirements.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn host_gateway_route_reports_ipv4_loopback_callback_listener() {
|
||||
let mut config = runtime_config();
|
||||
config.gateway_route = DockerGatewayRoute::HostGateway;
|
||||
config.gateway_callback_bind_address = Some("127.0.0.1:17670".parse().unwrap());
|
||||
let driver = test_driver_with_config(config);
|
||||
|
||||
let response = driver
|
||||
.get_gateway_listener_requirements(Request::new(GetGatewayListenerRequirementsRequest {}))
|
||||
.await
|
||||
.unwrap()
|
||||
.into_inner();
|
||||
|
||||
assert_eq!(response.requirements.len(), 1);
|
||||
assert_eq!(
|
||||
response.requirements[0].selector,
|
||||
Some(Selector::ExactBindAddress("127.0.0.1:17670".to_string()))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn container_visible_endpoint_rewrites_loopback_hosts() {
|
||||
assert_eq!(
|
||||
@@ -298,6 +323,31 @@ fn docker_gateway_route_uses_host_gateway_for_docker_desktop() {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn host_gateway_route_requests_ipv4_loopback_for_ipv6_primary() {
|
||||
assert_eq!(
|
||||
docker_gateway_callback_bind_address(
|
||||
&DockerGatewayRoute::HostGateway,
|
||||
"[::1]:17670".parse().unwrap(),
|
||||
),
|
||||
Some("127.0.0.1:17670".parse().unwrap())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn host_gateway_route_reuses_ipv4_primary_when_it_covers_loopback() {
|
||||
for primary in ["127.0.0.1:17670", "0.0.0.0:17670"] {
|
||||
assert_eq!(
|
||||
docker_gateway_callback_bind_address(
|
||||
&DockerGatewayRoute::HostGateway,
|
||||
primary.parse().unwrap(),
|
||||
),
|
||||
None,
|
||||
"{primary} already covers the IPv4 loopback callback"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn docker_gateway_route_uses_host_gateway_for_colima() {
|
||||
let info = SystemInfo {
|
||||
|
||||
@@ -275,13 +275,11 @@ and rootless pasta because the driver maps both local callback aliases to that
|
||||
literal. Other rootless helpers still fail closed. Podman Machine requests
|
||||
gateway loopback because its configured address is guest-visible and gvproxy
|
||||
terminates that route on host loopback. The gateway validates and binds every
|
||||
accepted callback listener. A callback address cannot equal the exact primary
|
||||
listener address because the gateway could not distinguish their authorization
|
||||
scopes. In particular, a Podman Machine gateway using the IPv4 loopback
|
||||
callback must place its primary listener on another address, such as IPv6
|
||||
loopback (`[::1]:17670`). Negotiated callback listeners expose only the
|
||||
gateway's sandbox-callable gRPC methods. Operator, health, reflection, and HTTP
|
||||
requests must use the primary listener.
|
||||
accepted callback requirement. If the primary listener covers the requested
|
||||
address, the gateway reuses it and relies on sandbox JWT authorization to limit
|
||||
the supervisor's RPCs. Otherwise, it creates an additional listener that
|
||||
exposes only the gateway's sandbox-callable gRPC methods. Operator, health,
|
||||
reflection, and HTTP requests must use the primary listener.
|
||||
|
||||
### Layer 3 Inner Sandbox Network Namespace
|
||||
|
||||
|
||||
@@ -103,7 +103,7 @@ fn gateway_listener_specs_with_default_route_ip(
|
||||
continue;
|
||||
};
|
||||
validate_gateway_listener_requirement(bind_address, requirement)?;
|
||||
add_callback_listener_spec(&mut specs, *address, requirement)?;
|
||||
add_callback_listener_spec(&mut specs, *address, requirement);
|
||||
}
|
||||
|
||||
for requirement in requirements {
|
||||
@@ -127,7 +127,7 @@ fn gateway_listener_specs_with_default_route_ip(
|
||||
}
|
||||
let address = SocketAddr::new(ip, bind_address.port());
|
||||
validate_resolved_gateway_listener(bind_address, address)?;
|
||||
add_callback_listener_spec(&mut specs, address, requirement)?;
|
||||
add_callback_listener_spec(&mut specs, address, requirement);
|
||||
}
|
||||
|
||||
for requirement in requirements {
|
||||
@@ -137,7 +137,7 @@ fn gateway_listener_specs_with_default_route_ip(
|
||||
validate_gateway_listener_requirement(bind_address, requirement)?;
|
||||
let address = SocketAddr::from(([127, 0, 0, 1], bind_address.port()));
|
||||
validate_resolved_gateway_listener(bind_address, address)?;
|
||||
add_callback_listener_spec(&mut specs, address, requirement)?;
|
||||
add_callback_listener_spec(&mut specs, address, requirement);
|
||||
}
|
||||
|
||||
Ok(specs)
|
||||
@@ -147,20 +147,17 @@ fn add_callback_listener_spec(
|
||||
specs: &mut Vec<GatewayListenerSpec>,
|
||||
address: SocketAddr,
|
||||
requirement: &GatewayListenerRequirement,
|
||||
) -> Result<()> {
|
||||
) {
|
||||
let scope = GatewayListenerScope::ComputeDriverCallback;
|
||||
if let Some(existing) = specs
|
||||
.iter_mut()
|
||||
.find(|existing| listener_covers(existing.address, address))
|
||||
{
|
||||
if existing.scope == GatewayListenerScope::Primary {
|
||||
return;
|
||||
}
|
||||
if existing.address == address {
|
||||
if existing.scope == GatewayListenerScope::Primary {
|
||||
return Err(Error::config(format!(
|
||||
"compute driver '{}' requested gateway callback listener {address}, but it is the same address as the primary listener; callback-only authorization cannot be preserved",
|
||||
requirement.driver_name()
|
||||
)));
|
||||
}
|
||||
return Ok(());
|
||||
return;
|
||||
}
|
||||
if !existing
|
||||
.covered_addresses
|
||||
@@ -171,10 +168,9 @@ fn add_callback_listener_spec(
|
||||
.covered_addresses
|
||||
.push(CoveredGatewayAddress { address, scope });
|
||||
}
|
||||
return Ok(());
|
||||
return;
|
||||
}
|
||||
specs.push(callback_listener_spec(address, requirement));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn callback_listener_spec(
|
||||
@@ -392,8 +388,8 @@ fn listener_covers(existing: SocketAddr, requested: SocketAddr) -> bool {
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{
|
||||
CoveredGatewayAddress, GatewayListenerProvenance, GatewayListenerScope,
|
||||
GatewayListenerSpec, bind_gateway_listeners, gateway_listener_specs,
|
||||
GatewayListenerProvenance, GatewayListenerScope, GatewayListenerSpec,
|
||||
bind_gateway_listeners, gateway_listener_specs,
|
||||
gateway_listener_specs_with_default_route_ip,
|
||||
};
|
||||
use crate::compute::GatewayListenerRequirement;
|
||||
@@ -402,7 +398,7 @@ mod tests {
|
||||
use tokio::net::TcpListener;
|
||||
|
||||
#[test]
|
||||
fn gateway_listener_specs_track_driver_address_covered_by_wildcard() {
|
||||
fn gateway_listener_specs_reuse_primary_when_wildcard_covers_driver_address() {
|
||||
let primary: SocketAddr = "0.0.0.0:8080".parse().unwrap();
|
||||
let docker: SocketAddr = "172.18.0.1:8080".parse().unwrap();
|
||||
let requirements = [
|
||||
@@ -412,20 +408,12 @@ mod tests {
|
||||
|
||||
assert_eq!(
|
||||
gateway_listener_specs(primary, &requirements).unwrap(),
|
||||
vec![GatewayListenerSpec {
|
||||
address: primary,
|
||||
scope: GatewayListenerScope::Primary,
|
||||
covered_addresses: vec![CoveredGatewayAddress {
|
||||
address: docker,
|
||||
scope: GatewayListenerScope::ComputeDriverCallback,
|
||||
}],
|
||||
provenance: None,
|
||||
}]
|
||||
vec![primary_listener_spec(primary)]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn gateway_listener_scope_for_local_addr_uses_covered_address_scope() {
|
||||
fn gateway_listener_scope_for_reused_primary_remains_primary() {
|
||||
let primary: SocketAddr = "0.0.0.0:8080".parse().unwrap();
|
||||
let docker: SocketAddr = "172.18.0.1:8080".parse().unwrap();
|
||||
let loopback: SocketAddr = "127.0.0.1:8080".parse().unwrap();
|
||||
@@ -436,7 +424,7 @@ mod tests {
|
||||
|
||||
assert_eq!(
|
||||
spec.scope_for_local_addr(docker),
|
||||
GatewayListenerScope::ComputeDriverCallback,
|
||||
GatewayListenerScope::Primary,
|
||||
);
|
||||
assert_eq!(
|
||||
spec.scope_for_local_addr(loopback),
|
||||
@@ -521,14 +509,14 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn gateway_listener_specs_track_podman_exact_when_primary_covers_it() {
|
||||
fn gateway_listener_specs_reuse_primary_when_it_covers_podman_exact() {
|
||||
let primary: SocketAddr = "0.0.0.0:8080".parse().unwrap();
|
||||
let podman_gateway: SocketAddr = "10.89.1.1:8080".parse().unwrap();
|
||||
|
||||
assert_eq!(
|
||||
gateway_listener_specs(primary, &[podman_listener_requirement(podman_gateway)],)
|
||||
.unwrap(),
|
||||
vec![primary_listener_spec_with_covered(primary, podman_gateway,)]
|
||||
vec![primary_listener_spec(primary)]
|
||||
);
|
||||
}
|
||||
|
||||
@@ -570,11 +558,9 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn gateway_listener_specs_track_default_route_when_primary_is_ipv4_wildcard() {
|
||||
fn gateway_listener_specs_reuse_ipv4_wildcard_for_default_route() {
|
||||
let primary: SocketAddr = "0.0.0.0:8080".parse().unwrap();
|
||||
let default_route_ip = "192.168.20.20".parse().unwrap();
|
||||
let callback = "192.168.20.20:8080".parse().unwrap();
|
||||
|
||||
assert_eq!(
|
||||
gateway_listener_specs_with_default_route_ip(
|
||||
primary,
|
||||
@@ -582,7 +568,7 @@ mod tests {
|
||||
Some(default_route_ip),
|
||||
)
|
||||
.unwrap(),
|
||||
vec![primary_listener_spec_with_covered(primary, callback)]
|
||||
vec![primary_listener_spec(primary)]
|
||||
);
|
||||
}
|
||||
|
||||
@@ -604,28 +590,23 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn gateway_listener_specs_track_podman_loopback_when_wildcard_primary_covers_it() {
|
||||
fn gateway_listener_specs_reuse_wildcard_primary_for_podman_loopback() {
|
||||
let primary = "0.0.0.0:8080".parse().unwrap();
|
||||
let loopback = "127.0.0.1:8080".parse().unwrap();
|
||||
|
||||
assert_eq!(
|
||||
gateway_listener_specs(primary, &[podman_loopback_listener_requirement()]).unwrap(),
|
||||
vec![primary_listener_spec_with_covered(primary, loopback)]
|
||||
vec![primary_listener_spec(primary)]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn gateway_listener_specs_reject_callback_matching_primary_address() {
|
||||
fn gateway_listener_specs_reuse_matching_primary_address() {
|
||||
let primary = "127.0.0.1:8080".parse().unwrap();
|
||||
|
||||
let err =
|
||||
gateway_listener_specs(primary, &[podman_loopback_listener_requirement()]).unwrap_err();
|
||||
|
||||
assert!(
|
||||
err.to_string()
|
||||
.contains("same address as the primary listener")
|
||||
assert_eq!(
|
||||
gateway_listener_specs(primary, &[podman_loopback_listener_requirement()]).unwrap(),
|
||||
vec![primary_listener_spec(primary)]
|
||||
);
|
||||
assert!(err.to_string().contains("callback-only authorization"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -742,21 +723,6 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
fn primary_listener_spec_with_covered(
|
||||
address: SocketAddr,
|
||||
covered_address: SocketAddr,
|
||||
) -> GatewayListenerSpec {
|
||||
GatewayListenerSpec {
|
||||
address,
|
||||
scope: GatewayListenerScope::Primary,
|
||||
covered_addresses: vec![CoveredGatewayAddress {
|
||||
address: covered_address,
|
||||
scope: GatewayListenerScope::ComputeDriverCallback,
|
||||
}],
|
||||
provenance: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn callback_listener_spec(
|
||||
address: SocketAddr,
|
||||
driver_name: &str,
|
||||
|
||||
@@ -38,7 +38,7 @@ For detailed driver behavior, refer to [Sandbox Compute Drivers](/reference/sand
|
||||
|
||||
On macOS, the install script uses Homebrew. The Homebrew package installs the `openshell` CLI, the gateway binary, and a Homebrew-managed gateway service.
|
||||
|
||||
The Homebrew service listens on `https://[::1]:17670` and generates a local mTLS bundle on install. The formula creates a Homebrew prefix config, such as `/opt/homebrew/var/openshell/gateway.toml`, with this IPv6 loopback default so Podman can use its separate IPv4 loopback callback listener. The gateway reads `~/.config/openshell/gateway.toml` instead when that file exists. Homebrew preserves existing prefix and user configs during upgrades.
|
||||
The Homebrew service uses the gateway's built-in `127.0.0.1:17670` listener and generates a local mTLS bundle on install. The installer registers `https://localhost:17670` with the CLI so TLS uses a DNS name covered by the generated certificate. The formula creates a Homebrew prefix config, such as `/opt/homebrew/var/openshell/gateway.toml`, without overriding `bind_address`. Docker Desktop and Podman Machine reuse the primary listener for sandbox callbacks when they can reach it. The gateway reads `~/.config/openshell/gateway.toml` instead when that file exists. Homebrew preserves user-edited prefix and user configs during upgrades; it removes the IPv6 bind only from an unchanged config generated by the affected formula.
|
||||
|
||||
The CLI reads the client bundle from `~/.config/openshell/gateways/openshell/mtls/`.
|
||||
|
||||
|
||||
@@ -47,7 +47,7 @@ Set these environment variables before starting the gateway:
|
||||
|
||||
For local access, the server certificate must be valid for the endpoint the CLI uses. Include `localhost`, `127.0.0.1`, and `::1` in the certificate SANs when users connect to a local gateway through loopback.
|
||||
|
||||
Package-managed local gateways generate this bundle automatically for the `openshell` gateway name. Homebrew uses `https://[::1]:17670` by default; Debian and RPM use `https://127.0.0.1:17670`.
|
||||
Package-managed local gateways generate this bundle automatically for the `openshell` gateway name. Homebrew registers `https://localhost:17670`; Debian and RPM use `https://127.0.0.1:17670`.
|
||||
When you register a package-managed local gateway with `openshell gateway add <endpoint> --local --name openshell`, the CLI refreshes its mTLS bundle from the package-managed TLS directory.
|
||||
On Homebrew, the gateway service also mirrors the Docker sandbox client bundle into `$HOME/.local/state/openshell/homebrew/tls` before startup so Docker Desktop can bind-mount the files into sandbox containers.
|
||||
|
||||
|
||||
@@ -31,7 +31,7 @@ Package-managed gateways do not require a TOML file. Create one at the package's
|
||||
|
||||
The Fedora/RHEL RPM template leaves `[openshell.gateway].bind_address` unset. The gateway therefore uses its built-in `127.0.0.1:17670` primary listener. The Podman driver negotiates separate, restricted listeners for sandbox callbacks, so the primary listener does not need a wildcard address. Set `bind_address` explicitly only when clients must reach the primary multiplexed API through another interface.
|
||||
|
||||
The Homebrew formula creates its prefix config once with `bind_address = "[::1]:17670"`. Keeping the primary API on IPv6 loopback leaves IPv4 loopback available for Podman Machine's sandbox callback listener. A user config takes precedence, and upgrades do not overwrite either config.
|
||||
The Homebrew formula creates its prefix config without setting `bind_address`, so the gateway uses its built-in `127.0.0.1:17670` primary listener. Docker Desktop and Podman Machine reuse that listener for sandbox callbacks. A user config takes precedence. Upgrades preserve user-edited configs and migrate only an unchanged prefix config generated with the affected IPv6-loopback default.
|
||||
|
||||
## Layout
|
||||
|
||||
|
||||
@@ -113,17 +113,15 @@ It overrides the gateway's configured default runtime class for that sandbox,
|
||||
while a typed `SandboxTemplate.runtime_class_name` value from the API still
|
||||
takes precedence.
|
||||
|
||||
Docker and Podman callback listeners accept only supervisor callback gRPC
|
||||
methods. Use the gateway's primary endpoint for CLI, administrator, health,
|
||||
reflection, inference-route management, and HTTP requests. A
|
||||
`PermissionDenied` response from one of the sandbox-visible callback addresses
|
||||
is expected for those requests. The gateway fails startup if a callback
|
||||
requirement resolves to the exact primary listener address because one socket
|
||||
cannot preserve both authorization scopes. For the IPv4-loopback callback used
|
||||
by Podman Machine, set `bind_address = "[::1]:17670"` for the primary listener
|
||||
and register `https://localhost:17670` as the CLI endpoint. The hostname matches
|
||||
the generated certificate and avoids the TLS transport error produced by a raw
|
||||
IPv6-literal endpoint. Do not broaden the primary listener to `0.0.0.0`.
|
||||
Docker and Podman report the address through which their sandboxes can reach
|
||||
the gateway. If the primary listener covers that address, the gateway reuses
|
||||
it and sandbox JWT authentication restricts the supervisor to its callback RPC
|
||||
allowlist. If the primary listener is not reachable through that address, the
|
||||
gateway creates an additional callback-only listener. Use the primary endpoint
|
||||
for CLI, administrator, health, reflection, inference-route management, and
|
||||
HTTP requests. A `PermissionDenied` response from an additional callback-only
|
||||
listener is expected for those requests. Do not broaden the primary listener
|
||||
to `0.0.0.0` solely to make sandbox callbacks reachable.
|
||||
|
||||
## Docker Driver
|
||||
|
||||
@@ -131,6 +129,11 @@ IPv6-literal endpoint. Do not broaden the primary listener to `0.0.0.0`.
|
||||
|
||||
The gateway talks to the Docker daemon to create sandbox containers. Docker is also required for local image builds from directories or Dockerfiles.
|
||||
|
||||
Docker Desktop and compatible macOS runtimes route `host.openshell.internal`
|
||||
through an IPv4 host-gateway alias. The gateway reuses an IPv4 primary listener
|
||||
that already covers loopback. Otherwise, the Docker driver requests a separate
|
||||
`127.0.0.1:<gateway-port>` callback-only listener.
|
||||
|
||||
For maintainer-level implementation details, refer to the [Docker driver README](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-docker/README.md).
|
||||
|
||||
Select Docker with `compute_drivers = ["docker"]` in `[openshell.gateway]`. Configure Docker driver values such as `socket_path`, `grpc_endpoint`, `network_name`, `supervisor_bin`, `supervisor_image`, `image_pull_policy`, `ssh_socket_path`, `sandbox_pids_limit`, and `guest_tls_*` in `[openshell.drivers.docker]`. When `socket_path` is unset, the driver uses the same responsive local socket selected by auto-detection. An explicitly selected Docker driver falls back to `/var/run/docker.sock` when no candidate responds.
|
||||
|
||||
+1
-1
@@ -470,7 +470,7 @@ detect_platform() {
|
||||
local_gateway_endpoint() {
|
||||
case "${PLATFORM:-$(detect_platform)}" in
|
||||
darwin)
|
||||
printf 'https://[::1]:%s\n' "$LOCAL_GATEWAY_PORT"
|
||||
printf 'https://localhost:%s\n' "$LOCAL_GATEWAY_PORT"
|
||||
;;
|
||||
*)
|
||||
printf 'https://127.0.0.1:%s\n' "$LOCAL_GATEWAY_PORT"
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
@@ -54,13 +55,23 @@ def test_generate_homebrew_formula_uses_tagged_macos_driver_asset_without_defaul
|
||||
assert "OPENSHELL_DRIVERS: " not in formula
|
||||
assert 'OPENSHELL_GATEWAY_CONFIG: "#{var}/openshell/gateway.toml"' not in formula
|
||||
assert "init-gateway-config.sh" not in formula
|
||||
assert 'bind_address = "127.0.0.1:17670"' not in formula
|
||||
assert 'gateway_config = var/"openshell/gateway.toml"' in formula
|
||||
assert "unless gateway_config.exist?" in formula
|
||||
generated_config = re.search(
|
||||
r"gateway_config_contents = <<~TOML\n(?P<contents>.*?)\n TOML",
|
||||
formula,
|
||||
flags=re.DOTALL,
|
||||
)
|
||||
assert generated_config is not None
|
||||
assert "[openshell.gateway]" in generated_config.group("contents")
|
||||
assert "bind_address =" not in generated_config.group("contents")
|
||||
assert 'bind_address = "[::1]:17670"' in formula
|
||||
assert "gateway_config.read == legacy_ipv6_gateway_config_contents" in formula
|
||||
assert "gateway_config.write gateway_config_contents" in formula
|
||||
assert '# compute_drivers = ["vm"]' not in formula
|
||||
assert (
|
||||
"openshell gateway add https://[::1]:17670 --local --name openshell" in formula
|
||||
"openshell gateway add https://localhost:17670 --local --name openshell"
|
||||
in formula
|
||||
)
|
||||
assert 'run opt_libexec/"openshell-gateway-homebrew-service"' in formula
|
||||
assert 'xdg_config_home="${XDG_CONFIG_HOME:-${HOME}/.config}"' in formula
|
||||
|
||||
@@ -341,14 +341,27 @@ class Openshell < Formula
|
||||
system bin/"openshell-gateway", "generate-certs", "--output-dir", var/"openshell/tls", "--server-san", "host.openshell.internal"
|
||||
|
||||
gateway_config = var/"openshell/gateway.toml"
|
||||
unless gateway_config.exist?
|
||||
gateway_config.write <<~TOML
|
||||
[openshell]
|
||||
version = 1
|
||||
gateway_config_contents = <<~TOML
|
||||
[openshell]
|
||||
version = 1
|
||||
|
||||
[openshell.gateway]
|
||||
bind_address = "[::1]:{LOCAL_GATEWAY_PORT}"
|
||||
TOML
|
||||
[openshell.gateway]
|
||||
TOML
|
||||
legacy_ipv6_gateway_config_contents = <<~TOML
|
||||
[openshell]
|
||||
version = 1
|
||||
|
||||
[openshell.gateway]
|
||||
bind_address = "[::1]:{LOCAL_GATEWAY_PORT}"
|
||||
TOML
|
||||
unless gateway_config.exist?
|
||||
gateway_config.write gateway_config_contents
|
||||
else
|
||||
# Migrate only the exact config generated by the affected formula. Keep
|
||||
# any user-edited config untouched.
|
||||
if gateway_config.read == legacy_ipv6_gateway_config_contents
|
||||
gateway_config.write gateway_config_contents
|
||||
end
|
||||
end
|
||||
|
||||
entitlements = var/"openshell/openshell-driver-vm.entitlements.plist"
|
||||
@@ -379,7 +392,7 @@ class Openshell < Formula
|
||||
brew services restart openshell
|
||||
|
||||
Register it with the OpenShell CLI:
|
||||
openshell gateway add https://[::1]:{LOCAL_GATEWAY_PORT} --local --name openshell
|
||||
openshell gateway add https://localhost:{LOCAL_GATEWAY_PORT} --local --name openshell
|
||||
EOS
|
||||
end
|
||||
|
||||
|
||||
@@ -100,8 +100,8 @@ assert_glibc_preflight_fails \
|
||||
"OpenShell Linux packages require glibc >= 2.28; detected musl or unsupported libc." \
|
||||
setup_ldd_musl
|
||||
|
||||
if [ "$(PLATFORM=darwin local_gateway_endpoint)" != "https://[::1]:17670" ]; then
|
||||
echo "FAIL: macOS local gateway endpoint must use IPv6 loopback" >&2
|
||||
if [ "$(PLATFORM=darwin local_gateway_endpoint)" != "https://localhost:17670" ]; then
|
||||
echo "FAIL: macOS local gateway endpoint must use a TLS-compatible loopback hostname" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
|
||||
Reference in New Issue
Block a user