feat(mxc): route provider credentials through host proxy

- pass effective provider policy and credential state into MXC sandbox creation
- inject revision-scoped placeholders into the MXC child environment
- resolve provider credentials only through governed host CONNECT proxy requests
- add gateway and driver coverage for secret isolation and policy propagation
- update MXC documentation and remove obsolete provider-v2 test setup

Signed-off-by: Akber Raza <akberr@nvidia.com>
This commit is contained in:
Akber Raza
2026-09-14 18:16:05 -07:00
committed by Drew Newberry
parent 01f41d93b0
commit 47b76ce7b6
9 changed files with 786 additions and 52 deletions
@@ -373,6 +373,18 @@ impl ProviderCredentialState {
.revision
}
/// Whether this snapshot contains endpoint-bound material that must be
/// resolved by a network proxy rather than exposed to the child process.
#[must_use]
pub fn requires_proxy_resolution(&self) -> bool {
let inner = self
.inner
.read()
.expect("provider credential state poisoned");
!inner.static_credential_bindings.is_empty()
|| !inner.current.dynamic_credentials.is_empty()
}
/// Remove a key from the credential snapshot's child env.
///
/// Used when a sandbox-side service (e.g., metadata server) fails to start
+28 -5
View File
@@ -7,9 +7,10 @@ OpenShell compute driver backed by **Microsoft MXC** (`wxc-exec`) on Windows.
This driver implements the gateway's ordinary in-process `ComputeDriver`
contract and is linked into `openshell-gateway`. It sets
`driver_reports_runtime_readiness`, so the gateway accepts driver-reported
readiness without a supervisor session. The canonical create-time
`SandboxPolicy` is carried by `DriverSandboxSpec.policy`. `process_container` launches a one-shot
AppContainer and is the default. The opt-in `isolation_session` backend uses the
readiness without a supervisor session. The gateway composes the create-time
effective `SandboxPolicy` and carries it on the driver-only copy of
`DriverSandboxSpec.policy`. `process_container` launches a one-shot AppContainer
and is the default. The opt-in `isolation_session` backend uses the
state-aware `provision` → `start` → `exec` → `stop` → `deprovision` lifecycle.
The driver launches and monitors the configured workload itself and self-reports
readiness; there is no in-sandbox supervisor or `ConnectSupervisor` relay.
@@ -22,6 +23,7 @@ readiness; there is no in-sandbox supervisor or `ConnectSupervisor` relay.
| UI policy | `process_container` advertises complete support and maps portable graphical UI, clipboard-direction, and input-injection controls to MXC; omitted fields inside an explicit section deny. `isolation_session` advertises no support, so the gateway rejects any explicit section before provisioning. | MXC support for persistent sessions |
| Governed egress (CONNECT proxy + OPA + L7) | Available behind `egress_proxy` on `process_container`; the driver starts a per-sandbox host CONNECT proxy, generates HTTPS MITM trust material, and injects the CA bundle into the sandbox process env | Gateway event-bus wiring follow-on |
| Network policy | Split into MXC `network.proxy` + trimmed OpenShell policy on `process_container`; `isolation_session` still rejects network config | MXC feedback item M1 for persistent sessions |
| Provider credentials | The child receives revision-scoped placeholders and non-secret provider environment only. The per-sandbox host proxy retains the resolver and substitutes credentials only for their bound endpoints. | — |
| Network middleware | ❌ rejected before launch because the MXC host proxy does not receive the gateway middleware registry | Gateway middleware-registry injection |
| Process policy (seccomp, uid/gid) | ❌ host-side governance design; OS isolation only | not pursued |
| Interactive exec/connect/forward | ❌ exec runs in-driver, no client attach | gateway interactive-exec surgery (follow-on) |
@@ -66,7 +68,15 @@ openshell sandbox create --name mxc-demo --policy demo.yaml `
--driver-config-json $config --env MODE=demo --no-tty
```
The `command` array is required and preserves Windows argument boundaries. `cwd` is optional. Environment variables come from the standard sandbox and template environment maps; the driver never copies values from the gateway host environment.
The `command` array is required and preserves Windows argument boundaries. `cwd`
is optional. Environment variables come from the standard sandbox and template
environment maps; the driver never copies values from the gateway host
environment. Provider-owned keys override matching entries case-insensitively.
MXC receives revision-scoped placeholders and explicitly classified non-secret
provider configuration, while credential values remain in the host proxy. When
governed egress is enabled, the driver replaces common TLS trust environment
variables with paths to the proxy-generated CA material and grants that
sandbox-unique CA directory read-write so the AppContainer can read it.
The host CONNECT proxy enforces network policy when governed egress is enabled.
Live policy replacement or merge updates remain unsupported; delete and recreate
@@ -117,7 +127,9 @@ The production driver maps the typed `SandboxPolicy` carried by the standard
driver request to MXC configuration before it inserts a registry entry or
invokes `wxc-exec`. Mapping failure therefore returns from `CreateSandbox`
without leaving a partial sandbox. There is no in-process policy side channel
or MXC-specific gateway composition variant.
or MXC-specific gateway composition variant. Provider resolver state uses a
separate, create-scoped in-process handoff because it intentionally cannot be
represented in the public compute-driver protobuf.
When `egress_proxy` is enabled, `EmbeddedPolicyMapper` uses `split_policy`
instead: MXC receives filesystem grants plus a loopback `network.proxy`
@@ -156,6 +168,17 @@ If governed egress is disabled, any network rule fails closed rather than launch
Parity and matrix tests under [`tests/`](tests/) cover the mapper on the Windows MSVC lane. The real-MXC lane also dry-runs every clipboard direction against the installed schema. The driver performs this mapping automatically; there is no separate policy-export command or example.
## Provider credential example
[`examples/run-provider-credential-test.ps1`](examples/run-provider-credential-test.ps1)
creates an MXC sandbox with an attached GitHub provider. Its policy explicitly
allows the graphical UI subsystem required by Windows PowerShell while denying
clipboard access and input injection; the existing policy mapper translates
that portable section to MXC's `ui` object. The probe verifies that the sandbox
sees a revision-scoped `GITHUB_TOKEN` placeholder, the host CONNECT proxy
substitutes it for `api.github.com`, and the same placeholder is rejected for a
different allowed endpoint.
## Packaging the demo for the demo box
Use [`examples/package-demo.ps1`](examples/package-demo.ps1) to assemble
@@ -243,7 +243,6 @@ try {
$env:OPENSHELL_GATEWAY = ""
Invoke-Cli @("gateway", "add", "http://127.0.0.1:$Port", "--local", "--name", $GatewayName) | Out-Null
Invoke-Cli @("gateway", "select", $GatewayName) | Out-Null
Invoke-Cli @("settings", "set", "--global", "--yes", "--key", "providers_v2_enabled", "--value", "true") | Out-Null
Invoke-Cli @("provider", "profile", "lint", "--file", $profileUsed) | Out-Null
Invoke-Cli @("provider", "profile", "import", "--file", $profileUsed) | Out-Null
Invoke-Cli @("provider", "create", "--name", $providerName, "--type", "mxc-github-e2e", "--credential", "GITHUB_TOKEN") | Out-Null
+249 -4
View File
@@ -15,12 +15,13 @@ use openshell_core::proto::compute::v1::{
WatchSandboxesPlatformEvent, WatchSandboxesSandboxEvent, watch_sandboxes_event,
};
use openshell_core::proto_struct::struct_to_json_value;
use openshell_core::provider_credentials::ProviderCredentialState;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::net::SocketAddr;
use std::path::PathBuf;
use std::pin::Pin;
use std::sync::Arc;
use std::sync::{Arc, Mutex as StdMutex};
use tokio::sync::{Mutex, broadcast, mpsc, watch};
use tokio::task::JoinHandle;
use tokio_stream::wrappers::ReceiverStream;
@@ -206,6 +207,10 @@ pub struct MxcComputeBackend {
registry: Arc<Mutex<HashMap<String, SandboxEntry>>>,
watch_tx: Arc<broadcast::Sender<WatchSandboxesEvent>>,
policy_mapper: Arc<dyn PolicyMapper>,
/// Provider resolver snapshots staged by the gateway immediately before
/// create. The driver consumes each entry exactly once; only placeholder
/// child environment values cross into MXC.
pending_provider_credentials: Arc<StdMutex<HashMap<String, ProviderCredentialState>>>,
/// In-process ETW → OCSF audit consumer (Plane A). `Some` only when
/// `config.etw_audit` is set and the session started; kept alive here so it
/// stops when the backend is dropped (held purely for its `Drop`, hence
@@ -282,6 +287,37 @@ fn sandbox_environment(sandbox: &DriverSandbox) -> Vec<String> {
environment
}
/// Merge provider-owned child environment values into MXC `process.env`.
///
/// Provider entries win case-insensitively, matching Windows environment
/// semantics. Secret values have already been replaced by revision-scoped
/// placeholders; explicitly classified GCP configuration is resolved by the
/// shared credential state because SDKs consume it before making a request.
fn append_provider_child_env(
env: &mut Vec<String>,
provider_credentials: Option<&ProviderCredentialState>,
) {
let Some(provider_credentials) = provider_credentials else {
return;
};
let mut provider_env = provider_credentials
.child_env_with_gcp_resolved()
.into_iter()
.collect::<Vec<_>>();
provider_env.sort_by(|(left, _), (right, _)| left.cmp(right));
env.retain(|entry| {
let key = entry.split_once('=').map_or(entry.as_str(), |(key, _)| key);
!provider_env
.iter()
.any(|(provider_key, _)| key.eq_ignore_ascii_case(provider_key))
});
env.extend(
provider_env
.into_iter()
.map(|(key, value)| format!("{key}={value}")),
);
}
fn configured_egress_addr(config: &MxcComputeConfig) -> Result<Option<SocketAddr>, tonic::Status> {
if !config.egress_proxy {
return Ok(None);
@@ -444,11 +480,19 @@ impl MxcComputeBackend {
// Production policy translation is always handled by the embedded
// mapper before any MXC lifecycle side effects begin.
policy_mapper: Arc::new(EmbeddedPolicyMapper),
pending_provider_credentials: Arc::new(StdMutex::new(HashMap::new())),
etw_session,
attribution,
}
}
/// Return the in-process create-time provider credential side channel.
pub fn provider_credentials_sink(
&self,
) -> Arc<StdMutex<HashMap<String, ProviderCredentialState>>> {
self.pending_provider_credentials.clone()
}
/// Test-only constructor wiring the in-process mock `wxc-exec` shim.
#[cfg(test)]
pub(crate) fn new_mocked(config: MxcComputeConfig) -> Self {
@@ -537,6 +581,14 @@ impl MxcComputeBackend {
pub async fn create_sandbox(&self, sandbox: &DriverSandbox) -> Result<(), tonic::Status> {
let sandbox_id = sandbox.id.clone();
// Consume before any fallible validation so rejected creates cannot
// retain real provider material in the staging map.
let provider_credentials = self
.pending_provider_credentials
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.remove(&sandbox_id);
Self::validate_sandbox_fields(sandbox)?;
let sandbox_config = sandbox_config(sandbox)?;
let (egress_addr, reserved_proxy_listener) = match configured_egress_addr(&self.config)? {
@@ -558,6 +610,15 @@ impl MxcComputeBackend {
// synchronously at the CreateSandbox boundary.
let policy = sandbox.spec.as_ref().and_then(|spec| spec.policy.as_ref());
let mapped = self.map_sandbox_policy(&sandbox_id, policy, egress_addr)?;
if provider_credentials
.as_ref()
.is_some_and(ProviderCredentialState::requires_proxy_resolution)
&& egress_addr.is_none()
{
return Err(tonic::Status::failed_precondition(
"mxc provider credentials require governed egress; enable egress_proxy so placeholders can be resolved by the host proxy",
));
}
if sandbox
.spec
@@ -628,6 +689,7 @@ impl MxcComputeBackend {
sandbox,
sandbox_config,
mapped,
provider_credentials,
reserved_proxy_listener,
startup_guard,
)
@@ -826,6 +888,7 @@ async fn run_lifecycle(
sandbox: DriverSandbox,
sandbox_config: MxcSandboxConfig,
mapped: MappedConfig,
provider_credentials: Option<ProviderCredentialState>,
mut reserved_proxy_listener: Option<std::net::TcpListener>,
_startup_guard: tokio::sync::OwnedMutexGuard<()>,
) {
@@ -845,7 +908,7 @@ async fn run_lifecycle(
sandbox_id: Some(sandbox_id.clone()),
sandbox_name: Some(sandbox_name.clone()),
openshell_endpoint: None,
provider_credentials: None,
provider_credentials: provider_credentials.clone(),
agent_proposals: openshell_core::proposals::AgentProposals::default(),
denial_tx: None,
activity_tx: None,
@@ -906,6 +969,7 @@ async fn run_lifecycle(
};
let command_line = encode_windows_command_line(&sandbox_config.command);
let mut environment = sandbox_environment(&sandbox);
append_provider_child_env(&mut environment, provider_credentials.as_ref());
append_tls_env_vars(&mut environment, host_proxy_ca_paths.as_ref());
info!(sandbox = %sandbox_name, count = environment.len(), "MXC process env vars");
let process = MxcProcess {
@@ -1235,8 +1299,9 @@ mod lifecycle_tests {
use futures::StreamExt;
use openshell_core::proto::compute::v1::{DriverSandboxSpec, DriverSandboxTemplate};
use openshell_core::proto::{
FilesystemPolicy, MiddlewareEndpointSelector, NetworkMiddlewareConfig, SandboxPolicy,
UiClipboardAccess, UiPolicy,
FilesystemPolicy, MiddlewareEndpointSelector, NetworkBinary, NetworkEndpoint,
NetworkMiddlewareConfig, NetworkPolicyRule, SandboxPolicy, StaticCredentialBinding,
StaticCredentialEndpointBinding, UiClipboardAccess, UiPolicy,
};
use std::time::Duration;
@@ -1292,6 +1357,32 @@ mod lifecycle_tests {
}
}
fn github_provider_credentials() -> ProviderCredentialState {
ProviderCredentialState::from_bound_environment(
42,
HashMap::from([(
"GITHUB_TOKEN".to_string(),
"raw-test-token-must-not-enter-mxc".to_string(),
)]),
HashMap::new(),
HashMap::new(),
HashMap::from([(
"GITHUB_TOKEN".to_string(),
StaticCredentialBinding {
endpoints: vec![StaticCredentialEndpointBinding {
host: "api.github.com".to_string(),
port: 443,
path: "/**".to_string(),
}],
credential_identity: "provider-github:GITHUB_TOKEN".to_string(),
workload_credential_handle: String::new(),
},
)]),
Vec::new(),
)
.expect("valid GitHub provider credential state")
}
fn with_policy(mut sandbox: DriverSandbox, policy: SandboxPolicy) -> DriverSandbox {
sandbox.spec.as_mut().unwrap().policy = Some(policy);
sandbox
@@ -1342,6 +1433,160 @@ mod lifecycle_tests {
assert_ne!(addr.port(), 0);
}
#[test]
fn governed_egress_rejects_non_loopback_and_isolation_session() {
let mut config = MxcComputeConfig {
egress_proxy: true,
egress_proxy_addr: "10.0.0.1:18080".into(),
..Default::default()
};
assert!(
configured_egress_addr(&config)
.unwrap_err()
.message()
.contains("127.0.0.1")
);
config.egress_proxy_addr = "127.0.0.1:18080".into();
config.backend = MxcBackend::IsolationSession;
assert!(
configured_egress_addr(&config)
.unwrap_err()
.message()
.contains("MXC M1")
);
}
#[test]
fn provider_child_environment_overrides_agent_values_with_placeholders() {
let credentials = github_provider_credentials();
let placeholder = credentials
.snapshot()
.child_env
.get("GITHUB_TOKEN")
.cloned()
.expect("GitHub placeholder");
let mut env = vec![
"github_token=agent-value".to_string(),
"UNCHANGED=value".to_string(),
];
append_provider_child_env(&mut env, Some(&credentials));
assert!(env.contains(&"UNCHANGED=value".to_string()));
assert!(!env.iter().any(|entry| entry == "github_token=agent-value"));
assert!(env.contains(&format!("GITHUB_TOKEN={placeholder}")));
assert!(!env.iter().any(|entry| entry.contains("raw-test-token")));
}
#[tokio::test]
async fn provider_state_requiring_resolution_fails_closed_without_governed_egress() {
let backend = MxcComputeBackend::new_mocked(MxcComputeConfig::default());
let sink = backend.provider_credentials_sink();
sink.lock()
.expect("provider credential staging lock poisoned")
.insert(
"sb-provider-no-proxy".to_string(),
github_provider_credentials(),
);
let error = backend
.create_sandbox(&with_policy(
driver_sandbox("sb-provider-no-proxy"),
fs_policy(&[]),
))
.await
.expect_err("provider credentials must require governed egress");
assert_eq!(error.code(), tonic::Code::FailedPrecondition);
assert!(error.message().contains("enable egress_proxy"));
assert!(
sink.lock()
.expect("provider credential staging lock poisoned")
.is_empty(),
"the driver must consume staged credential material on every create attempt"
);
}
#[tokio::test]
async fn governed_egress_puts_only_provider_placeholder_in_mxc_process_env() {
let mut config = MxcComputeConfig {
egress_proxy: true,
egress_proxy_addr: "127.0.0.1:18080".to_string(),
..Default::default()
};
config.backend = MxcBackend::ProcessContainer;
let backend = MxcComputeBackend::new_mocked(config);
let credentials = github_provider_credentials();
let placeholder = credentials
.snapshot()
.child_env
.get("GITHUB_TOKEN")
.cloned()
.expect("GitHub placeholder");
backend
.provider_credentials_sink()
.lock()
.expect("provider credential staging lock poisoned")
.insert("sb-provider-env".to_string(), credentials);
let mut policy = fs_policy(&[]);
policy.network_policies.insert(
"github".to_string(),
NetworkPolicyRule {
name: "github".to_string(),
endpoints: vec![NetworkEndpoint {
host: "api.github.com".to_string(),
port: 443,
protocol: "rest".to_string(),
provider_credentialed: true,
..Default::default()
}],
binaries: vec![NetworkBinary {
path: "cmd".to_string(),
..Default::default()
}],
},
);
let mut sandbox = with_policy(driver_sandbox("sb-provider-env"), policy);
sandbox
.spec
.as_mut()
.expect("sandbox spec")
.environment
.extend([
(
"GITHUB_TOKEN".to_string(),
"raw-agent-env-token".to_string(),
),
("UNCHANGED".to_string(), "value".to_string()),
]);
backend
.create_sandbox(&sandbox)
.await
.expect("create accepted");
wait_for(&backend, "sb-provider-env", |_| {
crate::mxc::mock_recorded_config("sb-provider-env").is_some()
})
.await
.expect("MXC config should be recorded");
let recorded =
crate::mxc::mock_recorded_config("sb-provider-env").expect("mock recorded config");
let env = recorded["process"]["env"]
.as_array()
.expect("MXC process env")
.iter()
.filter_map(serde_json::Value::as_str)
.collect::<Vec<_>>();
assert!(env.contains(&"UNCHANGED=value"));
assert!(env.contains(&format!("GITHUB_TOKEN={placeholder}").as_str()));
let encoded = recorded.to_string();
assert!(!encoded.contains("raw-agent-env-token"));
assert!(!encoded.contains("raw-test-token-must-not-enter-mxc"));
}
#[test]
fn sandbox_environment_inherits_host_with_spec_precedence() {
let mut sandbox = driver_sandbox("sb-env");
+7 -3
View File
@@ -159,10 +159,14 @@ impl openshell_server::ComputeDriverFactory for MxcFactory {
) -> openshell_core::Result<openshell_server::ComputeDriverInstance> {
let config: openshell_driver_mxc::MxcComputeConfig = context.driver_config()?;
let backend = openshell_driver_mxc::MxcComputeBackend::new(config);
let provider_credentials_sink = backend.provider_credentials_sink();
let driver = openshell_driver_mxc::ComputeDriverService::new(backend);
Ok(openshell_server::ComputeDriverInstance::InProcess(
std::sync::Arc::new(driver),
))
Ok(
openshell_server::ComputeDriverInstance::InProcessWithProviderCredentials {
driver: std::sync::Arc::new(driver),
provider_credentials_sink,
},
)
}
}
+276 -10
View File
@@ -35,9 +35,11 @@ use openshell_core::proto::compute::v1::{
gateway_listener_requirement::Selector, watch_sandboxes_event,
};
use openshell_core::proto::{
PlatformEvent, Sandbox, SandboxCondition, SandboxPhase, SandboxSpec, SandboxStatus,
SandboxTemplate, SandboxWorkloadTemplate, ServiceEndpoint, SshSession,
PlatformEvent, Sandbox, SandboxCondition, SandboxPhase, SandboxPolicy as ProtoSandboxPolicy,
SandboxSpec, SandboxStatus, SandboxTemplate, SandboxWorkloadTemplate, ServiceEndpoint,
SshSession,
};
use openshell_core::provider_credentials::ProviderCredentialState;
use openshell_core::telemetry::TelemetryComputeDriver;
use openshell_core::{ObjectLabels, ObjectWorkspace};
use prost::Message;
@@ -63,6 +65,60 @@ pub type DriverWatchStream =
Pin<Box<dyn Stream<Item = Result<WatchSandboxesEvent, Status>> + Send>>;
pub type SharedComputeDriver =
Arc<dyn ComputeDriver<WatchSandboxesStream = DriverWatchStream> + Send + Sync>;
pub type SandboxProviderCredentialsSink = Arc<StdMutex<HashMap<String, ProviderCredentialState>>>;
/// Driver-specific values that must be delivered atomically with sandbox
/// creation without expanding the public compute-driver protobuf contract.
#[derive(Clone, Default)]
pub struct SandboxCreateRuntimeInputs {
pub effective_policy: Option<ProtoSandboxPolicy>,
pub provider_credentials: Option<ProviderCredentialState>,
}
impl SandboxCreateRuntimeInputs {
#[must_use]
pub(crate) fn new(
effective_policy: ProtoSandboxPolicy,
provider_credentials: Option<ProviderCredentialState>,
) -> Self {
Self {
effective_policy: Some(effective_policy),
provider_credentials,
}
}
}
struct StagedProviderCredentials {
sink: SandboxProviderCredentialsSink,
sandbox_id: String,
}
impl StagedProviderCredentials {
fn stage(
sink: SandboxProviderCredentialsSink,
sandbox_id: &str,
credentials: ProviderCredentialState,
) -> Self {
let mut pending = sink
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
pending.insert(sandbox_id.to_string(), credentials);
drop(pending);
Self {
sink,
sandbox_id: sandbox_id.to_string(),
}
}
}
impl Drop for StagedProviderCredentials {
fn drop(&mut self) {
self.sink
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.remove(&self.sandbox_id);
}
}
use traced_driver::TracedDriver;
@@ -606,6 +662,7 @@ pub struct ComputeRuntime {
telemetry_compute_driver: TelemetryComputeDriver,
driver_process: Option<Arc<ManagedDriverProcess>>,
default_image: String,
provider_credentials_sink: Option<SandboxProviderCredentialsSink>,
store: Arc<Store>,
sandbox_index: SandboxIndex,
sandbox_watch_bus: SandboxWatchBus,
@@ -642,6 +699,7 @@ impl ComputeRuntime {
driver_name: String,
driver: SharedComputeDriver,
driver_process: Option<Arc<ManagedDriverProcess>>,
provider_credentials_sink: Option<SandboxProviderCredentialsSink>,
store: Arc<Store>,
sandbox_index: SandboxIndex,
sandbox_watch_bus: SandboxWatchBus,
@@ -739,6 +797,7 @@ impl ComputeRuntime {
telemetry_compute_driver: TelemetryComputeDriver::custom(),
driver_process,
default_image,
provider_credentials_sink,
store,
sandbox_index,
sandbox_watch_bus,
@@ -790,6 +849,7 @@ impl ComputeRuntime {
endpoint.name,
driver,
endpoint.driver_process,
None,
store,
sandbox_index,
sandbox_watch_bus,
@@ -804,6 +864,12 @@ impl ComputeRuntime {
&self.default_image
}
/// Whether this in-process driver accepts create-time provider state.
#[must_use]
pub(crate) fn accepts_create_time_provider_credentials(&self) -> bool {
self.provider_credentials_sink.is_some()
}
#[must_use]
pub fn driver_info_snapshots(&self) -> &[ComputeDriverInfoSnapshot] {
std::slice::from_ref(&self.driver_info)
@@ -909,9 +975,26 @@ impl ComputeRuntime {
}
pub async fn validate_sandbox_create(&self, sandbox: &Sandbox) -> Result<(), Status> {
self.validate_policy_capabilities(sandbox)?;
self.validate_sandbox_create_with_runtime_inputs(
sandbox,
&SandboxCreateRuntimeInputs::default(),
)
.await
}
pub(crate) async fn validate_sandbox_create_with_runtime_inputs(
&self,
sandbox: &Sandbox,
runtime_inputs: &SandboxCreateRuntimeInputs,
) -> Result<(), Status> {
self.validate_policy_capabilities(sandbox, runtime_inputs.effective_policy.as_ref())?;
let mut driver_sandbox = driver_sandbox_from_public(sandbox, &self.driver_info.name)
.map_err(|status| *status)?;
if let Some(effective_policy) = runtime_inputs.effective_policy.as_ref()
&& let Some(spec) = driver_sandbox.spec.as_mut()
{
spec.policy = Some(effective_policy.clone());
}
// Peek, never consume: create runs the same path immediately after and
// must still find the token.
if let Some(token) = take_staging_token(&mut driver_sandbox) {
@@ -934,11 +1017,13 @@ impl ComputeRuntime {
.map(|_| ())
}
fn validate_policy_capabilities(&self, sandbox: &Sandbox) -> Result<(), Status> {
let has_explicit_ui = sandbox
.spec
.as_ref()
.and_then(|spec| spec.policy.as_ref())
fn validate_policy_capabilities(
&self,
sandbox: &Sandbox,
effective_policy: Option<&ProtoSandboxPolicy>,
) -> Result<(), Status> {
let has_explicit_ui = effective_policy
.or_else(|| sandbox.spec.as_ref().and_then(|spec| spec.policy.as_ref()))
.and_then(|policy| policy.ui.as_ref())
.is_some();
if has_explicit_ui && !self.driver_info.supports_ui_policy {
@@ -955,10 +1040,32 @@ impl ComputeRuntime {
sandbox: Sandbox,
sandbox_token: Option<String>,
await_main_process_attachment: bool,
) -> Result<Sandbox, Status> {
self.create_sandbox_with_runtime_inputs(
sandbox,
sandbox_token,
await_main_process_attachment,
SandboxCreateRuntimeInputs::default(),
)
.await
}
pub(crate) async fn create_sandbox_with_runtime_inputs(
&self,
sandbox: Sandbox,
sandbox_token: Option<String>,
await_main_process_attachment: bool,
runtime_inputs: SandboxCreateRuntimeInputs,
) -> Result<Sandbox, Status> {
// Defense in depth for internal callers that bypass the public create
// handler's ValidateSandboxCreate step. This check has no side effects.
self.validate_policy_capabilities(&sandbox)?;
self.validate_policy_capabilities(&sandbox, runtime_inputs.effective_policy.as_ref())?;
if runtime_inputs.provider_credentials.is_some() && self.provider_credentials_sink.is_none()
{
return Err(Status::internal(
"provider credentials supplied to a compute driver without a create-time sink",
));
}
let sandbox_id = sandbox.object_id().to_string();
let mut sandbox = sandbox;
@@ -973,6 +1080,11 @@ impl ComputeRuntime {
let mut driver_sandbox = driver_sandbox_from_public(&sandbox, &self.driver_info.name)
.map_err(|status| *status)?;
if let Some(effective_policy) = runtime_inputs.effective_policy
&& let Some(spec) = driver_sandbox.spec.as_mut()
{
spec.policy = Some(effective_policy);
}
if let Some(staged) = staged.as_ref() {
set_rootfs_tar_path(&mut driver_sandbox, staged.path());
}
@@ -1022,6 +1134,18 @@ impl ComputeRuntime {
if let Some(spec) = driver_sandbox.spec.as_mut() {
spec.await_main_process_attachment = await_main_process_attachment;
}
let _staged_provider_credentials = match (
self.provider_credentials_sink.clone(),
runtime_inputs.provider_credentials,
) {
(Some(sink), Some(credentials)) => Some(StagedProviderCredentials::stage(
sink,
&sandbox_id,
credentials,
)),
(None, Some(_)) => unreachable!("provider credential sink checked before persistence"),
(_, None) => None,
};
match self
.driver
.call(
@@ -4949,6 +5073,7 @@ pub async fn new_test_runtime_with_driver(
telemetry_compute_driver: TelemetryComputeDriver::custom(),
driver_process: None,
default_image: "openshell/sandbox:test".to_string(),
provider_credentials_sink: None,
store,
sandbox_index: SandboxIndex::new(),
sandbox_watch_bus: SandboxWatchBus::new(),
@@ -5231,6 +5356,9 @@ mod tests {
workspace_rpcs_unimplemented: bool,
validate_create_calls: AtomicUsize,
create_calls: AtomicUsize,
provider_credentials_sink: Option<SandboxProviderCredentialsSink>,
provider_env_at_create: TestMutex<Option<HashMap<String, String>>>,
created_sandboxes: TestMutex<Vec<DriverSandbox>>,
}
#[tonic::async_trait]
@@ -5330,9 +5458,28 @@ mod tests {
async fn create_sandbox(
&self,
_request: Request<CreateSandboxRequest>,
request: Request<CreateSandboxRequest>,
) -> Result<tonic::Response<CreateSandboxResponse>, Status> {
self.create_calls.fetch_add(1, Ordering::Relaxed);
if let Some(sandbox) = request.into_inner().sandbox {
if let Some(sink) = &self.provider_credentials_sink {
let credentials = sink
.lock()
.expect("provider credential staging lock poisoned")
.get(&sandbox.id)
.cloned()
.expect("provider credentials must be staged before driver create");
*self
.provider_env_at_create
.lock()
.expect("provider env observation lock poisoned") =
Some(credentials.child_env_with_gcp_resolved());
}
self.created_sandboxes
.lock()
.expect("created sandbox observation lock poisoned")
.push(sandbox);
}
Ok(tonic::Response::new(CreateSandboxResponse {}))
}
@@ -5829,6 +5976,7 @@ mod tests {
telemetry_compute_driver: TelemetryComputeDriver::custom(),
driver_process: None,
default_image: "openshell/sandbox:test".to_string(),
provider_credentials_sink: None,
store,
sandbox_index: SandboxIndex::new(),
sandbox_watch_bus: SandboxWatchBus::new(),
@@ -5891,6 +6039,32 @@ mod tests {
);
}
#[tokio::test]
async fn effective_ui_policy_rejects_before_unsupported_driver_validation() {
let driver = Arc::new(TestDriver::default());
let runtime = test_runtime(driver.clone()).await;
let sandbox = sandbox_record(
"sb-effective-ui",
"effective-ui-policy",
SandboxPhase::Provisioning,
);
let runtime_inputs = SandboxCreateRuntimeInputs::new(
PublicSandboxPolicy {
ui: Some(UiPolicy::default()),
..Default::default()
},
None,
);
let error = runtime
.validate_sandbox_create_with_runtime_inputs(&sandbox, &runtime_inputs)
.await
.expect_err("an unsupported driver must reject the effective UI policy");
assert_eq!(error.code(), Code::InvalidArgument);
assert_eq!(driver.validate_create_calls.load(Ordering::Relaxed), 0);
}
#[tokio::test]
async fn explicit_ui_policy_reaches_driver_when_capability_is_complete() {
let driver = Arc::new(TestDriver::default());
@@ -5919,6 +6093,97 @@ mod tests {
assert_eq!(driver.validate_create_calls.load(Ordering::Relaxed), 1);
}
#[tokio::test]
async fn create_runtime_inputs_reach_driver_without_persisting_effective_policy_or_secrets() {
let sink: SandboxProviderCredentialsSink = Arc::new(TestMutex::new(HashMap::new()));
let driver = Arc::new(TestDriver {
provider_credentials_sink: Some(sink.clone()),
..Default::default()
});
let mut runtime = test_runtime(driver.clone()).await;
runtime.provider_credentials_sink = Some(sink.clone());
let mut sandbox = sandbox_record(
"sb-provider-inputs",
"provider-inputs",
SandboxPhase::Provisioning,
);
sandbox.spec = Some(SandboxSpec {
policy: Some(PublicSandboxPolicy {
version: 1,
..Default::default()
}),
..Default::default()
});
let provider_credentials = ProviderCredentialState::from_environment(
17,
HashMap::from([("GITHUB_TOKEN".to_string(), "raw-test-token".to_string())]),
HashMap::new(),
HashMap::new(),
);
let runtime_inputs = SandboxCreateRuntimeInputs::new(
PublicSandboxPolicy {
version: 2,
..Default::default()
},
Some(provider_credentials),
);
runtime
.create_sandbox_with_runtime_inputs(sandbox, None, false, runtime_inputs)
.await
.expect("create should succeed");
let observed_env = driver
.provider_env_at_create
.lock()
.expect("provider env observation lock poisoned")
.clone()
.expect("driver should observe staged provider credentials");
let observed_token = &observed_env["GITHUB_TOKEN"];
assert_ne!(observed_token, "raw-test-token");
assert!(observed_token.starts_with(openshell_core::secrets::PLACEHOLDER_PREFIX_PUBLIC));
assert!(
sink.lock()
.expect("provider credential staging lock poisoned")
.is_empty(),
"create-time credentials must be removed after the driver call"
);
let observed_policy_version = {
let created = driver
.created_sandboxes
.lock()
.expect("created sandbox observation lock poisoned");
created[0]
.spec
.as_ref()
.and_then(|spec| spec.policy.as_ref())
.map(|policy| policy.version)
};
assert_eq!(
observed_policy_version,
Some(2),
"the driver must receive the effective policy"
);
let persisted = runtime
.store
.get_message::<Sandbox>("sb-provider-inputs")
.await
.expect("persisted sandbox lookup should succeed")
.expect("sandbox should be persisted");
assert_eq!(
persisted
.spec
.as_ref()
.and_then(|spec| spec.policy.as_ref())
.map(|policy| policy.version),
Some(1),
"the public sandbox must retain its base policy"
);
}
async fn test_runtime_with_gateway_managed_lifecycle(
driver: SharedComputeDriver,
driver_name: &str,
@@ -10985,6 +11250,7 @@ mod tests {
"test-driver".to_string(),
Arc::new(TestDriver::default()),
None,
None,
store,
SandboxIndex::new(),
SandboxWatchBus::new(),
+178 -26
View File
@@ -1737,20 +1737,40 @@ async fn current_effective_policy_for_sandbox(
.as_ref()
.map(|spec| spec.providers.clone())
.unwrap_or_default();
let provider_records = super::provider::load_provider_environment_records(
state.store.as_ref(),
workspace,
&provider_names,
)
.await?;
current_effective_policy_for_sandbox_with_records(
state,
catalog,
sandbox,
sandbox_id,
&provider_records,
)
.await
}
async fn current_effective_policy_for_sandbox_with_records(
state: &ServerState,
catalog: &EffectiveProviderProfileCatalog,
sandbox: &Sandbox,
sandbox_id: &str,
provider_records: &[super::provider::ProviderEnvironmentRecord],
) -> Result<ProtoSandboxPolicy, Status> {
let global_settings = load_global_settings(state.store.as_ref()).await?;
if let Some(mut global_policy) = decode_policy_from_global_settings(&global_settings)? {
// A global policy replaces dynamic policy, but startup-only UI remains
// anchored to the sandbox spec so reads cannot misrepresent enforcement.
preserve_sandbox_startup_ui(&mut global_policy, sandbox);
return apply_effective_policy_context(
state,
return apply_effective_policy_context_from_records(
catalog,
workspace,
&provider_names,
provider_records,
global_policy,
PolicySource::Global,
)
.await;
);
}
let policy = if let Some(record) = state
@@ -1769,15 +1789,12 @@ async fn current_effective_policy_for_sandbox(
}
};
apply_effective_policy_context(
state,
apply_effective_policy_context_from_records(
catalog,
workspace,
&provider_names,
provider_records,
policy,
PolicySource::Sandbox,
)
.await
}
async fn effective_policy_for_source(
@@ -1812,17 +1829,26 @@ async fn apply_effective_policy_context(
catalog: &EffectiveProviderProfileCatalog,
workspace: &str,
provider_names: &[String],
mut policy: ProtoSandboxPolicy,
policy: ProtoSandboxPolicy,
policy_source: PolicySource,
) -> Result<ProtoSandboxPolicy, Status> {
clear_provider_credentialed_markers(&mut policy);
let mut provider_context = provider_policy_context_with_catalog(
let provider_records = super::provider::load_provider_environment_records(
state.store.as_ref(),
catalog,
workspace,
provider_names,
)
.await?;
apply_effective_policy_context_from_records(catalog, &provider_records, policy, policy_source)
}
fn apply_effective_policy_context_from_records(
catalog: &EffectiveProviderProfileCatalog,
provider_records: &[super::provider::ProviderEnvironmentRecord],
mut policy: ProtoSandboxPolicy,
policy_source: PolicySource,
) -> Result<ProtoSandboxPolicy, Status> {
clear_provider_credentialed_markers(&mut policy);
let mut provider_context = provider_policy_context_from_records(catalog, provider_records);
if !matches!(policy_source, PolicySource::Global) && !provider_context.layers.is_empty() {
policy = compose_effective_policy(&policy, &provider_context.layers);
}
@@ -2992,16 +3018,23 @@ async fn provider_policy_context_with_catalog(
workspace: &str,
provider_names: &[String],
) -> Result<ProviderPolicyContext, Status> {
let records =
super::provider::load_provider_environment_records(store, workspace, provider_names)
.await?;
Ok(provider_policy_context_from_records(catalog, &records))
}
fn provider_policy_context_from_records(
catalog: &EffectiveProviderProfileCatalog,
records: &[super::provider::ProviderEnvironmentRecord],
) -> ProviderPolicyContext {
let mut layers = Vec::new();
let mut credentialed_scopes = Vec::new();
let mut endpointless_provider_names = HashSet::new();
for name in provider_names {
let provider = store
.get_message_by_name::<Provider>(workspace, name)
.await
.map_err(|e| Status::internal(format!("failed to fetch provider '{name}': {e}")))?
.ok_or_else(|| Status::failed_precondition(format!("provider '{name}' not found")))?;
for record in records {
let name = &record.name;
let provider = &record.provider;
let provider_type = provider.r#type.trim();
let Some(profile) = super::provider::get_provider_type_profile_for_scope(
@@ -3017,7 +3050,7 @@ async fn provider_policy_context_with_catalog(
continue;
};
if !super::provider::provider_profile_endpoints_are_active(&profile, &provider) {
if !super::provider::provider_profile_endpoints_are_active(&profile, provider) {
endpointless_provider_names.insert(name.clone());
continue;
}
@@ -3045,11 +3078,11 @@ async fn provider_policy_context_with_catalog(
});
}
Ok(ProviderPolicyContext {
ProviderPolicyContext {
layers,
credentialed_scopes,
endpointless_provider_names,
})
}
}
fn endpoint_ports(endpoint: &NetworkEndpoint) -> Vec<u32> {
@@ -3221,6 +3254,125 @@ pub(super) async fn handle_get_gateway_config(
}))
}
/// Resolve the effective policy and provider credential snapshot required by
/// an in-process compute driver at sandbox creation time.
///
/// The policy, revision, endpoint bindings, and environment are all derived
/// from one immutable provider-record snapshot. Raw static credentials remain
/// in the returned resolver state; only revision-scoped placeholders are
/// exposed through its child environment.
pub(super) async fn resolve_sandbox_create_runtime_inputs(
state: &ServerState,
sandbox: &Sandbox,
) -> Result<crate::compute::SandboxCreateRuntimeInputs, Status> {
if !state.compute.accepts_create_time_provider_credentials() {
return Ok(crate::compute::SandboxCreateRuntimeInputs::default());
}
let sandbox_id = sandbox.object_id();
let workspace = sandbox.object_workspace();
let provider_names = sandbox
.spec
.as_ref()
.map(|spec| spec.providers.as_slice())
.unwrap_or_default();
let provider_profile_catalog = state
.provider_profile_sources
.snapshot_catalog(state.store.as_ref(), workspace)
.await?;
let provider_records = super::provider::load_provider_environment_records(
state.store.as_ref(),
workspace,
provider_names,
)
.await?;
let effective_policy = current_effective_policy_for_sandbox_with_records(
state,
&provider_profile_catalog,
sandbox,
sandbox_id,
&provider_records,
)
.await?;
let policy_credential_bindings =
policy_static_credential_endpoint_bindings(Some(&effective_policy))?;
validate_policy_credential_binding_context(
&provider_profile_catalog,
&provider_records,
&effective_policy,
&policy_credential_bindings,
)?;
let provider_env_revision = compute_provider_env_revision_from_records_and_policy_bindings(
&provider_profile_catalog,
&provider_records,
&policy_credential_bindings,
)?;
let mut provider_environment =
super::provider::resolve_provider_environment_from_records_with_policy_bindings_and_credentials(
state.store.as_ref(),
&provider_profile_catalog,
&provider_records,
&policy_credential_bindings,
&state.credentials,
Some(sandbox_id),
)
.await?;
// MXC uses the binding-capable host proxy. Withhold any static value that
// has no endpoint binding instead of exposing it directly to the process.
let unbound_static_keys = provider_environment
.static_credential_keys
.iter()
.filter(|key| {
!provider_environment
.static_credential_bindings
.contains_key(*key)
})
.cloned()
.collect::<Vec<_>>();
for key in unbound_static_keys {
warn!(
sandbox_id,
key = %key,
"withholding unbound static provider credential from MXC sandbox"
);
provider_environment.environment.remove(&key);
provider_environment.credential_expires_at_ms.remove(&key);
provider_environment.static_credential_keys.remove(&key);
}
let provider_credentials = if provider_records.is_empty() {
None
} else {
let non_secret_environment_keys = provider_environment
.environment
.keys()
.filter(|key| !provider_environment.static_credential_keys.contains(*key))
.cloned()
.collect();
Some(
openshell_core::provider_credentials::ProviderCredentialState::from_bound_environment(
provider_env_revision,
provider_environment.environment,
provider_environment.credential_expires_at_ms,
provider_environment.dynamic_credentials,
provider_environment.static_credential_bindings,
non_secret_environment_keys,
)
.map_err(|error| {
Status::failed_precondition(format!(
"invalid provider credential binding for sandbox '{sandbox_id}': {error}"
))
})?,
)
};
Ok(crate::compute::SandboxCreateRuntimeInputs::new(
effective_policy,
provider_credentials,
))
}
pub(super) async fn handle_get_sandbox_provider_environment(
state: &Arc<ServerState>,
request: Request<GetSandboxProviderEnvironmentRequest>,
@@ -3254,12 +3406,12 @@ pub(super) async fn handle_get_sandbox_provider_environment(
&provider_names,
)
.await?;
let effective_policy = current_effective_policy_for_sandbox(
let effective_policy = current_effective_policy_for_sandbox_with_records(
state.as_ref(),
&provider_profile_catalog,
&workspace,
&sandbox,
&sandbox_id,
&provider_records,
)
.await?;
let policy_credential_bindings =
+10 -2
View File
@@ -490,9 +490,12 @@ async fn handle_create_sandbox_inner(
)
.await?;
let runtime_inputs =
super::policy::resolve_sandbox_create_runtime_inputs(state.as_ref(), &sandbox).await?;
state
.compute
.validate_sandbox_create(&sandbox)
.validate_sandbox_create_with_runtime_inputs(&sandbox, &runtime_inputs)
.await
.map_err(|status| {
warn!(error = %status, "Rejecting sandbox create request");
@@ -518,7 +521,12 @@ async fn handle_create_sandbox_inner(
let sandbox = state
.compute
.create_sandbox(sandbox, sandbox_token, await_main_process_attachment)
.create_sandbox_with_runtime_inputs(
sandbox,
sandbox_token,
await_main_process_attachment,
runtime_inputs,
)
.await?;
info!(
+26 -1
View File
@@ -1047,13 +1047,19 @@ async fn terminate_signal() {
}
pub use compute::{
AcquiredRemoteDriverEndpoint, DriverWatchStream, ManagedDriverProcess, SharedComputeDriver,
AcquiredRemoteDriverEndpoint, DriverWatchStream, ManagedDriverProcess,
SandboxProviderCredentialsSink, SharedComputeDriver,
};
/// Driver instance returned by a compiled compute-driver factory.
pub enum ComputeDriverInstance {
/// A driver hosted in the gateway process.
InProcess(SharedComputeDriver),
/// An in-process driver with a create-time provider credential side channel.
InProcessWithProviderCredentials {
driver: SharedComputeDriver,
provider_credentials_sink: SandboxProviderCredentialsSink,
},
/// A driver process launched and owned by the gateway.
ManagedRemote(AcquiredRemoteDriverEndpoint),
}
@@ -1498,6 +1504,25 @@ async fn build_compute_runtime(
registration.name,
driver,
None,
None,
store,
sandbox_index,
sandbox_watch_bus,
tracing_log_bus,
supervisor_sessions,
)
.await
.map_err(|error| {
Error::execution(format!("failed to create compute runtime: {error}"))
})?,
ComputeDriverInstance::InProcessWithProviderCredentials {
driver,
provider_credentials_sink,
} => ComputeRuntime::from_driver(
registration.name,
driver,
None,
Some(provider_credentials_sink),
store,
sandbox_index,
sandbox_watch_bus,