mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-11-06 08:44:05 +08:00
feat(sandbox): support storage-preserving sandbox configuration updates (#4873)
* feat(gateway): add storage-preserving sandbox image updates Add image-only UpdateSandbox with conditional versions, durable replay, fenced pending intent, and Kubernetes storage retention. Enable non-destructive image repair after completed timeout cleanup and exercise both Kubernetes controller API lanes. Closes #4623 Signed-off-by: sjenning <493891+sjenning@users.noreply.github.com> * feat(gateway): add storage-preserving sandbox image updates OpenClaw-Publication: b2036b09-b697-4d65-aba9-5c96a3198f9c Signed-off-by: sjenning <493891+sjenning@users.noreply.github.com> * fix(gateway): preserve image update patch-release compatibility Move image updates to a separate authenticated service and retain the baseline public sandbox status. Persist image intent and revisions independently with atomic baseline-known lifecycle fencing and explicit post-upgrade activation. Preserve concurrent provisioning clocks, fence replaced attempts and delayed callbacks, and verify private-state transactions plus both Kubernetes controller API lanes. Signed-off-by: sjenning <493891+sjenning@users.noreply.github.com> * fix(gateway): preserve optional driver compatibility and repair CI Move image updates to a separate optional compute-driver service, preserving the baseline generated trait and streaming associated type. Add source and TCP/UDS transport regressions, capability negotiation coverage, and a default-unsupported in-process extension. Keep Go compatibility tests outside generated output and update production reflection expectations for the separate sandbox update service. Signed-off-by: sjenning <493891+sjenning@users.noreply.github.com> * fix(gateway): preserve baseline factory and descriptor APIs Restore the original in-process driver trait object and existing instance enum. Register optional image updates through a default factory hook without requiring changes to existing drivers or factories, and require registration plus capability before mutation. Accept OpenShell-only descriptors while retaining strict validation of present update services. Cover baseline registration, missing-updater no-write behavior, descriptor validation, and both live Kubernetes retention lanes. Signed-off-by: sjenning <493891+sjenning@users.noreply.github.com> * chore(ci): use upstream Kubernetes E2E defaults Drop the task-level fallback superseded by main's runner defaults and explicit e2e-conformance-only selector. Preserve current upstream task definitions and recovery waiver unchanged. Signed-off-by: sjenning <493891+sjenning@users.noreply.github.com> * fix(gateway): integrate image updates with backend protocol Adapt non-launch image updates to the upstream remote admission API while retaining resource-policy verification before forwarding. Cover both negotiated extensions and preservation of backend pins and persisted launch identity. Signed-off-by: sjenning <493891+sjenning@users.noreply.github.com> * fix(gateway): remove sandbox image state on deletion Signed-off-by: Gordon Sim <gsim@redhat.com> * feat(gateway): expand UpdateSandbox configuration updates Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(sandbox): recover pending updates and address review feedback Signed-off-by: Gordon Sim <gsim@redhat.com> * perf(server): index pending sandbox updates for recovery Signed-off-by: Gordon Sim <gsim@redhat.com> * docs(sandbox): clarify stop behavior when updates are disabled Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(server): scope deletion bypass to the matching sandbox update Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(server): back off pending sandbox update recovery Signed-off-by: Gordon Sim <gsim@redhat.com> * ci: restore combined-chart Kubernetes e2e coverage Signed-off-by: Gordon Sim <gsim@redhat.com> * test(kubernetes): use uuid for update request IDs Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(kubernetes): reject oversized annotations before accepting sandbox updates Signed-off-by: Gordon Sim <gsim@redhat.com> * docs(skills): correct pending sandbox update recovery guidance Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(sandbox): preserve terminal results when stopping missing compute Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(server): bound sandbox update attempts to unblock recovery and deletion Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(server): remove unintended sandbox label count limit Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(kubernetes): check known pod names during sandbox update preflight Signed-off-by: Gordon Sim <gsim@redhat.com> * docs: clarify sandbox update permissions and driver support Signed-off-by: Gordon Sim <gsim@redhat.com> --------- Signed-off-by: sjenning <493891+sjenning@users.noreply.github.com> Signed-off-by: Gordon Sim <gsim@redhat.com> Co-authored-by: sjenning <493891+sjenning@users.noreply.github.com>
This commit is contained in:
@@ -421,6 +421,30 @@ above) instead of relying only on `OPENSHELL_GATEWAY_ENDPOINT`; some tests call
|
||||
### Kubernetes E2E Notes
|
||||
|
||||
Use `mise run e2e:kubernetes` for the standard Helm-backed Kubernetes suite.
|
||||
Set `OPENSHELL_E2E_KUBE_TEST=kubernetes_image_update` to scope the Rust suite to
|
||||
combined image, command, environment, and metadata updates with retained sandbox,
|
||||
SSH, and PVC identities. The next start verifies the new argv and environment. This target is
|
||||
included in both existing Agent Sandbox API-version lanes; unlike the Python
|
||||
Docker suite, it exercises the driver that supports image updates.
|
||||
|
||||
Image updates default to disabled outside test fixtures. The e2e wrapper opts
|
||||
in only for a fresh gateway fixture that it owns; set
|
||||
`OPENSHELL_E2E_KUBE_IMAGE_UPDATES=0` to prevent that opt-in. Do not propagate
|
||||
activation to an existing shared gateway or enable it during a mixed-version
|
||||
rollout. For manual testing, follow the retirement and activation checklist in
|
||||
`docs/kubernetes/high-availability.mdx` before setting
|
||||
`gatewayConfig: {"openshell.gateway": {enable_sandbox_updates: true}}`.
|
||||
The gateway process equivalents are `--enable-sandbox-updates true` and
|
||||
`OPENSHELL_ENABLE_SANDBOX_UPDATES=true`. Retire every old process and Job,
|
||||
settle outstanding driver work, prevent old binaries returning, and do not
|
||||
treat PostgreSQL connection markers or membership TTL expiry as retirement
|
||||
proof. SQLite testing supports one gateway process only. Image-update clients
|
||||
must call the separate `SandboxUpdates` service on the existing gateway
|
||||
listener, not add a method to the baseline `OpenShell` service or inspect new
|
||||
public `SandboxStatus` image fields. Resume a pending operation with
|
||||
`SandboxUpdates.UpdateSandbox`, the current resource version, and no selected
|
||||
fields, following the documented request-ID semantics.
|
||||
|
||||
The kube e2e wrapper creates only one port-forward, to `svc/openshell`; it no
|
||||
longer forwards the unauthenticated health listener or runs a `/readyz` e2e
|
||||
target. `/readyz` remains covered by server unit/integration tests.
|
||||
@@ -611,3 +635,13 @@ for dependencies still declared in `Chart.yaml`.
|
||||
| `deploy/kube/manifests/envoy-gateway-openshell.yaml` | GatewayClass and BackendTrafficPolicy for Envoy Gateway (`mise run helm:gateway:apply`) |
|
||||
| `tasks/scripts/helm-k3s-local.sh` | k3d cluster create/delete/start/stop/status |
|
||||
| `tasks/scripts/keycloak-k8s-setup.sh` | Keycloak deploy, realm import, and development TLS trust anchor |
|
||||
|
||||
## Sandbox update regression lane
|
||||
|
||||
Run `mise run e2e:kubernetes:updates` for the testkit update suite against both
|
||||
supported Agent Sandbox APIs with the combined Helm chart and its bundled
|
||||
workspace resources. It verifies stopped-only updates, retained CR/PVC
|
||||
identity and contents, command/environment changes, and request replay.
|
||||
`OPENSHELL_E2E_UPDATE_ORIGINAL_IMAGE` and `OPENSHELL_E2E_UPDATE_TARGET_IMAGE` can
|
||||
select mirrored Alpine 3.21 and 3.22 manifests. Defaults use the public registry
|
||||
cache; keep immutable digests when overriding them.
|
||||
|
||||
Generated
+1
@@ -4576,6 +4576,7 @@ dependencies = [
|
||||
"prost",
|
||||
"prost-types",
|
||||
"rcgen",
|
||||
"regex",
|
||||
"rustls",
|
||||
"serde",
|
||||
"serde_json",
|
||||
|
||||
@@ -4116,6 +4116,14 @@ pub async fn sandbox_stop(
|
||||
.into_inner()
|
||||
.sandbox
|
||||
.ok_or_else(|| miette!("gateway returned no sandbox after stop"))?;
|
||||
// A successful stop can preserve a canonical result when the transient
|
||||
// compute resource is already gone. There is no Stopped event to wait for.
|
||||
if sandbox.phase() == SandboxPhase::Completed as i32
|
||||
|| (sandbox.phase() == SandboxPhase::Error as i32 && has_main_process_result(&sandbox))
|
||||
{
|
||||
println!("{} Sandbox {name} already finished", "✓".green().bold());
|
||||
return Ok(());
|
||||
}
|
||||
wait_for_lifecycle_phase(&mut client, sandbox, SandboxPhase::Stopped).await?;
|
||||
println!("{} Stopped sandbox {name}", "✓".green().bold());
|
||||
Ok(())
|
||||
|
||||
@@ -67,6 +67,7 @@ struct SandboxState {
|
||||
create_requests: Arc<Mutex<Vec<CreateSandboxRequest>>>,
|
||||
expose_service_requests: Arc<Mutex<Vec<openshell_core::proto::ExposeServiceRequest>>>,
|
||||
fail_delete_sandbox_message: Arc<Mutex<Option<String>>>,
|
||||
stop_response: Arc<Mutex<Option<Sandbox>>>,
|
||||
vm_error_after_started: Arc<AtomicBool>,
|
||||
vm_error_with_observed_exit: Arc<AtomicBool>,
|
||||
vm_slow_progress_before_ready: Arc<AtomicBool>,
|
||||
@@ -231,7 +232,17 @@ impl OpenShell for TestOpenShell {
|
||||
&self,
|
||||
_request: tonic::Request<openshell_core::proto::StopSandboxRequest>,
|
||||
) -> Result<Response<SandboxResponse>, Status> {
|
||||
Err(Status::unimplemented("unused"))
|
||||
let sandbox = self
|
||||
.state
|
||||
.stop_response
|
||||
.lock()
|
||||
.await
|
||||
.clone()
|
||||
.ok_or_else(|| Status::unimplemented("unused"))?;
|
||||
Ok(Response::new(SandboxResponse {
|
||||
sandbox: Some(sandbox),
|
||||
..Default::default()
|
||||
}))
|
||||
}
|
||||
|
||||
async fn start_sandbox(
|
||||
@@ -1709,6 +1720,57 @@ fn test_config() -> run::SandboxCreateConfig<'static> {
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn sandbox_stop_accepts_terminal_results_without_waiting_for_stopped() {
|
||||
let server = run_server().await;
|
||||
let tls = test_tls(&server);
|
||||
let config_dir = helpers::tempdir().unwrap();
|
||||
for (phase, exit_code, reason, succeeds) in [
|
||||
(
|
||||
SandboxPhase::Completed,
|
||||
Some(0),
|
||||
"MainProcessCompleted",
|
||||
true,
|
||||
),
|
||||
(SandboxPhase::Error, Some(17), "MainProcessFailed", true),
|
||||
(SandboxPhase::Stopped, None, "Stopped", true),
|
||||
(
|
||||
SandboxPhase::Error,
|
||||
Some(17),
|
||||
"ComputeResourceMissing",
|
||||
false,
|
||||
),
|
||||
] {
|
||||
let sandbox = Sandbox {
|
||||
status: Some(SandboxStatus {
|
||||
phase: phase as i32,
|
||||
exit_code,
|
||||
conditions: vec![SandboxCondition {
|
||||
r#type: "Ready".into(),
|
||||
status: "False".into(),
|
||||
reason: reason.into(),
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
*server.openshell.state.stop_response.lock().await = Some(sandbox);
|
||||
Environment::new()
|
||||
.set("XDG_CONFIG_HOME", config_dir.path())
|
||||
.run_async(async {
|
||||
let result = tokio::time::timeout(
|
||||
Duration::from_secs(5),
|
||||
run::sandbox_stop(&server.endpoint, "terminal-stop-test", "default", &tls),
|
||||
)
|
||||
.await
|
||||
.expect("stop must finish without watching for another phase");
|
||||
assert_eq!(result.is_ok(), succeeds, "{reason}: {result:?}");
|
||||
})
|
||||
.await;
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn sandbox_delete_continues_after_entry_failure() {
|
||||
let server = run_server().await;
|
||||
|
||||
@@ -65,6 +65,20 @@ configuration validation live in `openshell_core::middleware`. Policy admission
|
||||
and the supervisor runtime consume the same contract without introducing a
|
||||
dependency from the policy crate to the supervisor implementation.
|
||||
|
||||
## Optional Compute Updates
|
||||
|
||||
The generated `proto::compute::v1::compute_driver_server::ComputeDriver` trait
|
||||
retains its baseline methods and streaming associated type. Supporting external
|
||||
drivers additionally serve `ComputeDriverConfigurationUpdates` on the same
|
||||
endpoint and advertise `openshell.compute.update-sandbox-configuration`.
|
||||
The request carries image and lifecycle preconditions plus optional command and
|
||||
environment replacements. Drivers without the capability need no new method.
|
||||
|
||||
Linked drivers opt in through `openshell_core::compute::ComputeDriver` and
|
||||
`ComputeDriverFactory::build_with_configuration_updates`. The default factory
|
||||
hook calls `build` without an updater. Registration and capability negotiation
|
||||
are both required. Metadata-only updates are handled by the gateway.
|
||||
|
||||
## Protobuf Struct Conversion
|
||||
|
||||
Use `openshell_core::proto_struct` when crossing between `serde_json` values and
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
//! Opt-in in-process sandbox updates, separate from the baseline driver contract.
|
||||
|
||||
use crate::proto::compute::v1::compute_driver_server;
|
||||
|
||||
/// Optional in-process configuration updater, registered separately by a driver factory.
|
||||
///
|
||||
/// Existing drivers need only the generated baseline trait, not this extension.
|
||||
/// Supporting factories opt in through `ComputeDriverFactory::build_with_configuration_updates`
|
||||
/// and advertise the configuration-update capability. External servers use the
|
||||
/// separate generated `ComputeDriverConfigurationUpdates` service on the same endpoint.
|
||||
#[tonic::async_trait]
|
||||
pub trait ComputeDriver: compute_driver_server::ComputeDriver {
|
||||
async fn update_sandbox_configuration(
|
||||
&self,
|
||||
_request: tonic::Request<crate::proto::compute::v1::UpdateSandboxConfigurationRequest>,
|
||||
) -> Result<
|
||||
tonic::Response<crate::proto::compute::v1::UpdateSandboxConfigurationResponse>,
|
||||
tonic::Status,
|
||||
> {
|
||||
Err(tonic::Status::unimplemented(
|
||||
"sandbox configuration updates are not supported by this driver",
|
||||
))
|
||||
}
|
||||
}
|
||||
@@ -252,6 +252,11 @@ pub struct Config {
|
||||
/// Disabled for local gateways by default.
|
||||
pub enable_websocket_tunnel: bool,
|
||||
|
||||
/// Operator activation after all gateways and outstanding legacy driver
|
||||
/// operations have been drained onto the configuration-update-compatible protocol.
|
||||
/// Disabled by default; not a license to run mixed-version writers.
|
||||
pub enable_sandbox_updates: bool,
|
||||
|
||||
/// Disabled-by-default gateway interceptor service configs.
|
||||
pub gateway_interceptors: Vec<GatewayInterceptorConfig>,
|
||||
|
||||
@@ -999,6 +1004,7 @@ impl Config {
|
||||
oidc: None,
|
||||
auth: GatewayAuthConfig::default(),
|
||||
enable_websocket_tunnel: false,
|
||||
enable_sandbox_updates: false,
|
||||
gateway_interceptors: Vec::new(),
|
||||
provider_profile_sources: vec![GatewayProviderProfileSourceConfig::User],
|
||||
mtls_auth: MtlsAuthConfig::default(),
|
||||
|
||||
@@ -25,6 +25,10 @@ pub const COMPUTE_ISOLATION_BACKEND: &str = "openshell.compute.isolation-backend
|
||||
/// Driver implements `SuspendSandbox` and resuming through `StartSandbox`.
|
||||
pub const COMPUTE_SUSPEND_RESUME: &str = "openshell.compute.suspend-resume";
|
||||
|
||||
/// Optional stopped-state workload configuration updates retaining the resource and storage.
|
||||
pub const COMPUTE_UPDATE_SANDBOX_CONFIGURATION: &str =
|
||||
"openshell.compute.update-sandbox-configuration";
|
||||
|
||||
const MAX_IMPLEMENTATION_NAME_BYTES: usize = 128;
|
||||
const MAX_IMPLEMENTATION_VERSION_BYTES: usize = 128;
|
||||
const MAX_CAPABILITY_BYTES: usize = 128;
|
||||
@@ -128,6 +132,7 @@ pub fn gateway_metadata(family: ExtensionFamily) -> PeerMetadata {
|
||||
if family == ExtensionFamily::Compute {
|
||||
supported_capabilities.push(COMPUTE_LAUNCH_AUTHENTICATION.to_string());
|
||||
supported_capabilities.push(COMPUTE_ISOLATION_BACKEND.to_string());
|
||||
supported_capabilities.push(COMPUTE_UPDATE_SANDBOX_CONFIGURATION.to_string());
|
||||
}
|
||||
PeerMetadata {
|
||||
protocol_version: Some(ProtocolVersion {
|
||||
@@ -399,6 +404,23 @@ fn valid_capability(value: &str) -> bool {
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn compute_gateway_advertises_both_optional_extensions() {
|
||||
let gateway = gateway_metadata(ExtensionFamily::Compute);
|
||||
assert!(
|
||||
gateway
|
||||
.supported_capabilities
|
||||
.iter()
|
||||
.any(|c| c == COMPUTE_ISOLATION_BACKEND)
|
||||
);
|
||||
assert!(
|
||||
gateway
|
||||
.supported_capabilities
|
||||
.iter()
|
||||
.any(|c| c == COMPUTE_UPDATE_SANDBOX_CONFIGURATION)
|
||||
);
|
||||
}
|
||||
|
||||
fn compatible() -> (PeerMetadata, PeerMetadata) {
|
||||
let gateway = gateway_metadata(ExtensionFamily::Compute);
|
||||
let extension = extension_metadata(
|
||||
|
||||
@@ -11,6 +11,7 @@
|
||||
|
||||
pub mod activity;
|
||||
pub mod auth;
|
||||
pub mod compute;
|
||||
pub mod config;
|
||||
pub mod container_paths;
|
||||
pub mod denial;
|
||||
@@ -58,6 +59,7 @@ pub mod rpc_error;
|
||||
pub mod sandbox_env;
|
||||
pub mod sandbox_generation;
|
||||
pub mod sandbox_session;
|
||||
pub mod sandbox_update;
|
||||
pub mod secrets;
|
||||
pub mod settings;
|
||||
pub mod shell;
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
//! Shared projections for selected stopped-sandbox configuration changes.
|
||||
use crate::proto::{Sandbox, SandboxCommandUpdate, SandboxMapUpdate};
|
||||
use std::collections::HashMap;
|
||||
|
||||
/// Gateway-owned identity and lifecycle metadata is never caller-replaceable.
|
||||
/// Other keys, including user keys under `OpenShell` domains, remain editable.
|
||||
#[must_use]
|
||||
pub fn is_system_metadata_key(key: &str) -> bool {
|
||||
key.starts_with("internal.openshell.ai/")
|
||||
|| matches!(
|
||||
key,
|
||||
"openshell.nvidia.com/retention"
|
||||
| "openshell.ai/managed-by"
|
||||
| "openshell.ai/sandbox-id"
|
||||
| "openshell.ai/sandbox-name"
|
||||
| "openshell.ai/sandbox-namespace"
|
||||
| "openshell.ai/sandbox-workspace"
|
||||
| "openshell.ai/gateway-id"
|
||||
| "openshell.ai/caller-driver-config-used"
|
||||
| "openshell.ai/resource-admission-identities"
|
||||
)
|
||||
}
|
||||
|
||||
fn replace_user_metadata(current: &mut HashMap<String, String>, update: &SandboxMapUpdate) {
|
||||
current.retain(|key, _| is_system_metadata_key(key));
|
||||
current.extend(update.values.clone());
|
||||
}
|
||||
|
||||
/// Caller must validate syntax and forbid system keys before projecting.
|
||||
pub fn apply_configuration(
|
||||
sandbox: &mut Sandbox,
|
||||
command: Option<&SandboxCommandUpdate>,
|
||||
environment: Option<&SandboxMapUpdate>,
|
||||
labels: Option<&SandboxMapUpdate>,
|
||||
annotations: Option<&SandboxMapUpdate>,
|
||||
) {
|
||||
if let Some(spec) = sandbox.spec.as_mut() {
|
||||
if let Some(command) = command {
|
||||
spec.command.clone_from(&command.argv);
|
||||
}
|
||||
if let Some(environment) = environment {
|
||||
spec.environment.clone_from(&environment.values);
|
||||
}
|
||||
}
|
||||
if let Some(metadata) = sandbox.metadata.as_mut() {
|
||||
if let Some(labels) = labels {
|
||||
replace_user_metadata(&mut metadata.labels, labels);
|
||||
}
|
||||
if let Some(annotations) = annotations {
|
||||
replace_user_metadata(&mut metadata.annotations, annotations);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn replacement_preserves_owned_keys_without_freezing_user_domains() {
|
||||
let mut metadata = HashMap::from([
|
||||
("internal.openshell.ai/auth-epoch".into(), "trusted".into()),
|
||||
("openshell.nvidia.com/retention".into(), "ephemeral".into()),
|
||||
("openshell.ai/user-note".into(), "old".into()),
|
||||
("agents.x-k8s.io/user-note".into(), "old".into()),
|
||||
]);
|
||||
replace_user_metadata(&mut metadata, &SandboxMapUpdate::default());
|
||||
assert_eq!(metadata.len(), 2);
|
||||
assert_eq!(metadata["openshell.nvidia.com/retention"], "ephemeral");
|
||||
assert!(!is_system_metadata_key("openshell.ai/user-note"));
|
||||
assert!(!is_system_metadata_key("agents.x-k8s.io/user-note"));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,306 @@
|
||||
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
//! Baseline server source fixture adapted for the upstream suspend RPC.
|
||||
//! It deliberately implements only the baseline generated `ComputeDriver` methods,
|
||||
//! including its concrete streaming associated type, with no image-update method
|
||||
//! or in-process extension trait. New optional RPCs must not break this source.
|
||||
|
||||
use openshell_core::proto::compute::v1::{
|
||||
AuthenticateSandboxRequest, AuthenticateSandboxResponse, CreateSandboxRequest,
|
||||
CreateSandboxResponse, DeleteSandboxRequest, DeleteSandboxResponse, DeleteWorkspaceRequest,
|
||||
DeleteWorkspaceResponse, EnsureWorkspaceRequest, EnsureWorkspaceResponse,
|
||||
GetCapabilitiesRequest, GetCapabilitiesResponse, GetSandboxRequest, GetSandboxResponse,
|
||||
ListSandboxesRequest, ListSandboxesResponse, StartSandboxRequest, StartSandboxResponse,
|
||||
StopSandboxRequest, StopSandboxResponse, SuspendSandboxRequest, SuspendSandboxResponse,
|
||||
UpdateSandboxConfigurationRequest, UpdateSandboxConfigurationResponse,
|
||||
ValidateSandboxCreateRequest, ValidateSandboxCreateResponse, WatchSandboxesEvent,
|
||||
WatchSandboxesRequest,
|
||||
compute_driver_client::ComputeDriverClient,
|
||||
compute_driver_configuration_updates_client::ComputeDriverConfigurationUpdatesClient,
|
||||
compute_driver_configuration_updates_server::{
|
||||
ComputeDriverConfigurationUpdates, ComputeDriverConfigurationUpdatesServer,
|
||||
},
|
||||
compute_driver_server::{ComputeDriver, ComputeDriverServer},
|
||||
};
|
||||
use prost::Message;
|
||||
use std::pin::Pin;
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::task::{Context, Poll};
|
||||
use tokio::net::{TcpListener, TcpStream};
|
||||
use tokio_stream::Stream;
|
||||
use tonic::{Request, Response, Status};
|
||||
|
||||
#[derive(Debug)]
|
||||
struct BaselineDriver;
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl ComputeDriver for BaselineDriver {
|
||||
async fn get_capabilities(
|
||||
&self,
|
||||
_request: Request<GetCapabilitiesRequest>,
|
||||
) -> Result<Response<GetCapabilitiesResponse>, Status> {
|
||||
Ok(Response::new(GetCapabilitiesResponse {
|
||||
driver_name: "baseline-source-fixture".into(),
|
||||
..Default::default()
|
||||
}))
|
||||
}
|
||||
|
||||
async fn authenticate_sandbox(
|
||||
&self,
|
||||
_request: Request<AuthenticateSandboxRequest>,
|
||||
) -> Result<Response<AuthenticateSandboxResponse>, Status> {
|
||||
Err(Status::unimplemented("fixture"))
|
||||
}
|
||||
|
||||
async fn validate_sandbox_create(
|
||||
&self,
|
||||
_request: Request<ValidateSandboxCreateRequest>,
|
||||
) -> Result<Response<ValidateSandboxCreateResponse>, Status> {
|
||||
Err(Status::unimplemented("fixture"))
|
||||
}
|
||||
|
||||
async fn get_sandbox(
|
||||
&self,
|
||||
_request: Request<GetSandboxRequest>,
|
||||
) -> Result<Response<GetSandboxResponse>, Status> {
|
||||
Err(Status::unimplemented("fixture"))
|
||||
}
|
||||
|
||||
async fn list_sandboxes(
|
||||
&self,
|
||||
_request: Request<ListSandboxesRequest>,
|
||||
) -> Result<Response<ListSandboxesResponse>, Status> {
|
||||
Ok(Response::new(ListSandboxesResponse { sandboxes: vec![] }))
|
||||
}
|
||||
|
||||
async fn create_sandbox(
|
||||
&self,
|
||||
_request: Request<CreateSandboxRequest>,
|
||||
) -> Result<Response<CreateSandboxResponse>, Status> {
|
||||
Err(Status::unimplemented("fixture"))
|
||||
}
|
||||
|
||||
async fn stop_sandbox(
|
||||
&self,
|
||||
_request: Request<StopSandboxRequest>,
|
||||
) -> Result<Response<StopSandboxResponse>, Status> {
|
||||
Err(Status::unimplemented("fixture"))
|
||||
}
|
||||
|
||||
async fn suspend_sandbox(
|
||||
&self,
|
||||
_request: Request<SuspendSandboxRequest>,
|
||||
) -> Result<Response<SuspendSandboxResponse>, Status> {
|
||||
Err(Status::unimplemented("fixture"))
|
||||
}
|
||||
|
||||
async fn start_sandbox(
|
||||
&self,
|
||||
_request: Request<StartSandboxRequest>,
|
||||
) -> Result<Response<StartSandboxResponse>, Status> {
|
||||
Err(Status::unimplemented("fixture"))
|
||||
}
|
||||
|
||||
async fn delete_sandbox(
|
||||
&self,
|
||||
_request: Request<DeleteSandboxRequest>,
|
||||
) -> Result<Response<DeleteSandboxResponse>, Status> {
|
||||
Err(Status::unimplemented("fixture"))
|
||||
}
|
||||
|
||||
type WatchSandboxesStream = tokio_stream::Empty<Result<WatchSandboxesEvent, Status>>;
|
||||
|
||||
async fn watch_sandboxes(
|
||||
&self,
|
||||
_request: Request<WatchSandboxesRequest>,
|
||||
) -> Result<Response<Self::WatchSandboxesStream>, Status> {
|
||||
Ok(Response::new(tokio_stream::empty()))
|
||||
}
|
||||
|
||||
async fn ensure_workspace(
|
||||
&self,
|
||||
_request: Request<EnsureWorkspaceRequest>,
|
||||
) -> Result<Response<EnsureWorkspaceResponse>, Status> {
|
||||
Err(Status::unimplemented("fixture"))
|
||||
}
|
||||
|
||||
async fn delete_workspace(
|
||||
&self,
|
||||
_request: Request<DeleteWorkspaceRequest>,
|
||||
) -> Result<Response<DeleteWorkspaceResponse>, Status> {
|
||||
Err(Status::unimplemented("fixture"))
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Default)]
|
||||
struct ImageUpdates {
|
||||
calls: Arc<Mutex<Vec<UpdateSandboxConfigurationRequest>>>,
|
||||
}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl ComputeDriverConfigurationUpdates for ImageUpdates {
|
||||
async fn update_sandbox_configuration(
|
||||
&self,
|
||||
request: Request<UpdateSandboxConfigurationRequest>,
|
||||
) -> Result<Response<UpdateSandboxConfigurationResponse>, Status> {
|
||||
self.calls.lock().unwrap().push(request.into_inner());
|
||||
Ok(Response::new(UpdateSandboxConfigurationResponse {}))
|
||||
}
|
||||
}
|
||||
|
||||
struct TcpIncoming(TcpListener);
|
||||
|
||||
impl Stream for TcpIncoming {
|
||||
type Item = std::io::Result<TcpStream>;
|
||||
|
||||
fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
|
||||
self.0.poll_accept(cx).map(|result| {
|
||||
Some(result.map(|(stream, _)| {
|
||||
openshell_core::net::set_tcp_nodelay_best_effort(&stream);
|
||||
stream
|
||||
}))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
async fn roundtrip(serve_image_updates: bool) {
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let address = listener.local_addr().unwrap();
|
||||
let extension = ImageUpdates::default();
|
||||
let optional_service = serve_image_updates
|
||||
.then(|| ComputeDriverConfigurationUpdatesServer::new(extension.clone()));
|
||||
let (shutdown_tx, shutdown_rx) = tokio::sync::oneshot::channel();
|
||||
let server = tokio::spawn(async move {
|
||||
tonic::transport::Server::builder()
|
||||
.add_service(ComputeDriverServer::new(BaselineDriver))
|
||||
.add_optional_service(optional_service)
|
||||
.serve_with_incoming_shutdown(TcpIncoming(listener), async {
|
||||
let _ = shutdown_rx.await;
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
});
|
||||
let channel = tonic::transport::Endpoint::from_shared(format!("http://{address}"))
|
||||
.unwrap()
|
||||
.connect()
|
||||
.await
|
||||
.unwrap();
|
||||
let mut baseline = ComputeDriverClient::new(channel.clone());
|
||||
assert_eq!(
|
||||
baseline
|
||||
.get_capabilities(GetCapabilitiesRequest::default())
|
||||
.await
|
||||
.unwrap()
|
||||
.into_inner()
|
||||
.driver_name,
|
||||
"baseline-source-fixture"
|
||||
);
|
||||
assert!(
|
||||
baseline
|
||||
.list_sandboxes(ListSandboxesRequest {})
|
||||
.await
|
||||
.unwrap()
|
||||
.into_inner()
|
||||
.sandboxes
|
||||
.is_empty()
|
||||
);
|
||||
assert!(
|
||||
baseline
|
||||
.watch_sandboxes(WatchSandboxesRequest {})
|
||||
.await
|
||||
.unwrap()
|
||||
.into_inner()
|
||||
.message()
|
||||
.await
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
|
||||
let request = UpdateSandboxConfigurationRequest {
|
||||
preconditions: Some(
|
||||
openshell_core::proto::compute::v1::SandboxUpdatePreconditions {
|
||||
sandbox_id: "sandbox-id".into(),
|
||||
image: "example/workload:v2".into(),
|
||||
..Default::default()
|
||||
},
|
||||
),
|
||||
..Default::default()
|
||||
};
|
||||
let result = ComputeDriverConfigurationUpdatesClient::new(channel)
|
||||
.update_sandbox_configuration(request.clone())
|
||||
.await;
|
||||
if serve_image_updates {
|
||||
result.unwrap();
|
||||
assert_eq!(*extension.calls.lock().unwrap(), vec![request]);
|
||||
} else {
|
||||
assert_eq!(result.unwrap_err().code(), tonic::Code::Unimplemented);
|
||||
assert!(extension.calls.lock().unwrap().is_empty());
|
||||
}
|
||||
// Absence of the optional service must not impair the original service.
|
||||
baseline
|
||||
.list_sandboxes(ListSandboxesRequest {})
|
||||
.await
|
||||
.unwrap();
|
||||
drop(baseline);
|
||||
shutdown_tx.send(()).unwrap();
|
||||
server.await.unwrap();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn baseline_generated_server_source_and_transport_remain_compatible() {
|
||||
roundtrip(false).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn optional_image_service_shares_the_baseline_channel() {
|
||||
roundtrip(true).await;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn baseline_compute_service_methods_are_frozen() {
|
||||
let descriptor =
|
||||
prost_types::FileDescriptorSet::decode(openshell_core::FILE_DESCRIPTOR_SET).unwrap();
|
||||
let compute = descriptor
|
||||
.file
|
||||
.iter()
|
||||
.find(|file| file.package.as_deref() == Some("openshell.compute.v1"))
|
||||
.unwrap();
|
||||
let baseline = compute
|
||||
.service
|
||||
.iter()
|
||||
.find(|service| service.name.as_deref() == Some("ComputeDriver"))
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
baseline
|
||||
.method
|
||||
.iter()
|
||||
.map(|method| method.name.as_deref().unwrap())
|
||||
.collect::<Vec<_>>(),
|
||||
vec![
|
||||
"GetCapabilities",
|
||||
"AuthenticateSandbox",
|
||||
"ValidateSandboxCreate",
|
||||
"GetSandbox",
|
||||
"ListSandboxes",
|
||||
"CreateSandbox",
|
||||
"StopSandbox",
|
||||
"StartSandbox",
|
||||
"SuspendSandbox",
|
||||
"DeleteSandbox",
|
||||
"WatchSandboxes",
|
||||
"EnsureWorkspace",
|
||||
"DeleteWorkspace",
|
||||
]
|
||||
);
|
||||
let image_updates = compute
|
||||
.service
|
||||
.iter()
|
||||
.find(|service| service.name.as_deref() == Some("ComputeDriverConfigurationUpdates"))
|
||||
.unwrap();
|
||||
assert_eq!(image_updates.method.len(), 1);
|
||||
assert_eq!(
|
||||
image_updates.method[0].name.as_deref(),
|
||||
Some("UpdateSandboxConfiguration")
|
||||
);
|
||||
}
|
||||
@@ -3156,6 +3156,9 @@ fn validate_docker_outer_fence(
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
#[tonic::async_trait]
|
||||
impl openshell_core::compute::ComputeDriver for ComputeDriverService {}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl ComputeDriver for ComputeDriverService {
|
||||
type WatchSandboxesStream = WatchStream;
|
||||
@@ -3318,6 +3321,9 @@ impl ComputeDriver for ComputeDriverService {
|
||||
}
|
||||
}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl openshell_core::compute::ComputeDriver for DockerComputeDriver {}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl ComputeDriver for DockerComputeDriver {
|
||||
async fn authenticate_sandbox(
|
||||
|
||||
@@ -43,6 +43,7 @@ tracing-subscriber = { workspace = true }
|
||||
thiserror = { workspace = true }
|
||||
miette = { workspace = true }
|
||||
notify = "8"
|
||||
regex = { workspace = true }
|
||||
rcgen = { workspace = true }
|
||||
bytes = { workspace = true }
|
||||
http = { workspace = true }
|
||||
|
||||
@@ -302,6 +302,36 @@ storage. The driver confirms the stop from both the published `Suspended`
|
||||
condition and deletion of the backing pod. Legacy `v1alpha1` controllers omit
|
||||
a usable stopped condition, so pod deletion alone confirms their stop.
|
||||
|
||||
The optional `openshell.compute.update-sandbox-configuration` capability supports
|
||||
image, command, and environment updates while stopped. External drivers serve
|
||||
`ComputeDriverConfigurationUpdates` separately from the baseline driver service;
|
||||
linked factories opt in through `build_with_configuration_updates`.
|
||||
The request identifies the sandbox, original resource UID, expected image,
|
||||
operation ID, and monotonic revision. The driver checks the suspended state,
|
||||
completed bootstrap, current resource UID, and absence of workload and supervisor
|
||||
Pods by directly reading the controller-recorded workload name, the Sandbox
|
||||
resource name, and the stable dedicated-supervisor name. Duplicate names are
|
||||
read once, and terminating Pods block updates. These checks make at most three
|
||||
Pod reads regardless of namespace size; they do not discover unexpectedly named
|
||||
Pods. The driver revalidates
|
||||
external-resource admission before patching the existing resource with a
|
||||
`resourceVersion` condition. The patch includes the verified UID; Kubernetes
|
||||
immutable-UID validation also rejects a same-name replacement. No resource or
|
||||
PVC is recreated. Existing workspace contents are not reseeded.
|
||||
|
||||
A retry verifies durable operation evidence and the live selected targets before
|
||||
returning success without another patch. Different parameters or older revisions
|
||||
fail closed, including when later operations return to an earlier image.
|
||||
|
||||
The service reconciles workload argv and effective user environment with the image
|
||||
in the same conditional patch. The
|
||||
main-process annotation and agent environment supply the next runtime generation;
|
||||
the trusted container launcher and runtime environment remain intact. Retry
|
||||
records identify selected fields and verify their live target values without
|
||||
copying environment values into operation evidence. Top-level public sandbox
|
||||
labels and annotations remain gateway metadata; this operation does not change
|
||||
Kubernetes pod metadata.
|
||||
|
||||
The workspace PVC size defaults to `workspace_default_storage_size`. Set
|
||||
`workspace_storage_class` to pin the PVC to a specific `StorageClass`; an empty
|
||||
value omits `storageClassName` so the cluster's default `StorageClass` applies.
|
||||
|
||||
@@ -90,6 +90,9 @@ use tokio::sync::{OnceCell, mpsc};
|
||||
use tokio_stream::wrappers::ReceiverStream;
|
||||
use tracing::{debug, info, warn};
|
||||
|
||||
#[path = "image_update.rs"]
|
||||
mod image_update;
|
||||
|
||||
pub type WatchStream =
|
||||
Pin<Box<dyn Stream<Item = Result<WatchSandboxesEvent, KubernetesDriverError>> + Send>>;
|
||||
|
||||
@@ -1341,7 +1344,10 @@ impl KubernetesComputeDriver {
|
||||
openshell_core::extension_protocol::ExtensionFamily::Compute,
|
||||
"openshell/kubernetes",
|
||||
openshell_core::VERSION,
|
||||
[],
|
||||
[
|
||||
openshell_core::extension_protocol::COMPUTE_UPDATE_SANDBOX_CONFIGURATION
|
||||
.to_string(),
|
||||
],
|
||||
)),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -337,6 +337,38 @@ impl ComputeDriver for ComputeDriverService {
|
||||
}
|
||||
}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl openshell_core::compute::ComputeDriver for ComputeDriverService {
|
||||
async fn update_sandbox_configuration(
|
||||
&self,
|
||||
request: Request<openshell_core::proto::compute::v1::UpdateSandboxConfigurationRequest>,
|
||||
) -> Result<
|
||||
Response<openshell_core::proto::compute::v1::UpdateSandboxConfigurationResponse>,
|
||||
Status,
|
||||
> {
|
||||
self.rpc_tracer
|
||||
.trace(openshell_otel::rpc::UPDATE_SANDBOX_CONFIGURATION, async {
|
||||
self.driver
|
||||
.update_sandbox_configuration(&request.into_inner())
|
||||
.await
|
||||
.map_err(|error| {
|
||||
Status::from(openshell_core::ComputeDriverError::from(error))
|
||||
})?;
|
||||
Ok(Response::new(
|
||||
openshell_core::proto::compute::v1::UpdateSandboxConfigurationResponse {},
|
||||
))
|
||||
})
|
||||
.await
|
||||
}
|
||||
}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl openshell_core::proto::compute::v1::compute_driver_configuration_updates_server::ComputeDriverConfigurationUpdates for ComputeDriverService {
|
||||
async fn update_sandbox_configuration(&self, request: Request<openshell_core::proto::compute::v1::UpdateSandboxConfigurationRequest>) -> Result<Response<openshell_core::proto::compute::v1::UpdateSandboxConfigurationResponse>, Status> {
|
||||
openshell_core::compute::ComputeDriver::update_sandbox_configuration(self, request).await
|
||||
}
|
||||
}
|
||||
|
||||
fn workspace_delete_requires_namespace_access(mode: WorkspaceMode) -> bool {
|
||||
matches!(mode, WorkspaceMode::Managed)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,614 @@
|
||||
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
//! Storage-preserving, stopped-only workload image mutation.
|
||||
|
||||
use super::{
|
||||
KUBE_API_TIMEOUT, KubernetesComputeDriver, KubernetesDriverError, LABEL_SANDBOX_NAME,
|
||||
SANDBOX_POD_NAME_ANNOTATION, SANDBOX_VERSION_V1BETA1, WORKSPACE_INIT_CONTAINER_NAME,
|
||||
accepts_auth_namespace, admission_error, annotation_or_label,
|
||||
is_kube_resource_version_conflict, parse_kubernetes_runtime_identity, sandbox_id_from_object,
|
||||
sandbox_runtime_bootstrap_in_progress, select_expected_sandbox_runtime,
|
||||
};
|
||||
use crate::sandbox_runtime::SandboxRuntimeNames;
|
||||
use k8s_openapi::api::core::v1::Pod;
|
||||
use kube::Api;
|
||||
use kube::api::{DynamicObject, ListParams, Patch, PatchParams};
|
||||
use openshell_core::proto::compute::v1::SandboxUpdatePreconditions;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
use std::sync::LazyLock;
|
||||
|
||||
// Deliberately separate from the runtime bootstrap operation annotations. Those
|
||||
// annotations drive reconciliation and only describe create/start/stop.
|
||||
const IMAGE_UPDATE_ANNOTATION: &str = "openshell.ai/sandbox-image-update";
|
||||
// Kubernetes limits the sum of all annotation keys and values, in bytes.
|
||||
const MAX_ANNOTATION_BYTES: usize = 256 * 1024;
|
||||
|
||||
#[derive(Debug, Deserialize, Serialize, PartialEq, Eq)]
|
||||
struct ImageUpdateRecord {
|
||||
operation_id: String,
|
||||
expected_image: String,
|
||||
image: String,
|
||||
revision: u64,
|
||||
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
|
||||
command_updated: bool,
|
||||
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
|
||||
environment_updated: bool,
|
||||
}
|
||||
|
||||
impl KubernetesComputeDriver {
|
||||
/// Update only a suspended Sandbox's workload and workspace seed images.
|
||||
/// The resource, claim templates, existing claims, and runtime bootstrap
|
||||
/// annotations are never replaced. A retry is recognized from durable
|
||||
/// operation evidence on the same immutable Sandbox resource.
|
||||
#[tracing::instrument(
|
||||
name = "kubernetes.update_sandbox_image",
|
||||
skip_all,
|
||||
fields(
|
||||
otel.name = "kubernetes.update_sandbox_image",
|
||||
otel.status_code = tracing::field::Empty,
|
||||
sandbox.id = %request.sandbox_id,
|
||||
)
|
||||
)]
|
||||
#[cfg(test)]
|
||||
pub async fn update_sandbox_image(
|
||||
&self,
|
||||
request: &SandboxUpdatePreconditions,
|
||||
) -> Result<(), KubernetesDriverError> {
|
||||
self.update_sandbox_configuration(
|
||||
&openshell_core::proto::compute::v1::UpdateSandboxConfigurationRequest {
|
||||
preconditions: Some(request.clone()),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
pub async fn update_sandbox_configuration(
|
||||
&self,
|
||||
request: &openshell_core::proto::compute::v1::UpdateSandboxConfigurationRequest,
|
||||
) -> Result<(), KubernetesDriverError> {
|
||||
let image = request.preconditions.as_ref().ok_or_else(|| {
|
||||
KubernetesDriverError::InvalidArgument("preconditions are required".into())
|
||||
})?;
|
||||
validate_configuration(request)?;
|
||||
self.update_workload(image, Some(request)).await
|
||||
}
|
||||
|
||||
async fn update_workload(
|
||||
&self,
|
||||
request: &SandboxUpdatePreconditions,
|
||||
configuration: Option<
|
||||
&openshell_core::proto::compute::v1::UpdateSandboxConfigurationRequest,
|
||||
>,
|
||||
) -> Result<(), KubernetesDriverError> {
|
||||
let span_status = openshell_otel::ErrorStatusGuard::current();
|
||||
validate_image_update_request(request)?;
|
||||
let _guard = self
|
||||
.lifecycle_gates
|
||||
.gate_for(&request.sandbox_id)
|
||||
.lock_owned()
|
||||
.await;
|
||||
// Include discovery, admission and Pod reads in a single bound.
|
||||
let result = tokio::time::timeout(
|
||||
KUBE_API_TIMEOUT,
|
||||
self.update_sandbox_image_inner(request, configuration),
|
||||
)
|
||||
.await
|
||||
.unwrap_or_else(|_| {
|
||||
Err(KubernetesDriverError::Message(
|
||||
"timed out updating Kubernetes sandbox image; retry the same operation".into(),
|
||||
))
|
||||
});
|
||||
span_status.finish(result)
|
||||
}
|
||||
|
||||
async fn update_sandbox_image_inner(
|
||||
&self,
|
||||
request: &SandboxUpdatePreconditions,
|
||||
configuration: Option<
|
||||
&openshell_core::proto::compute::v1::UpdateSandboxConfigurationRequest,
|
||||
>,
|
||||
) -> Result<(), KubernetesDriverError> {
|
||||
let expected = parse_kubernetes_runtime_identity(&request.expected_runtime_identity)?;
|
||||
self.wait_for_operator_allowlist()
|
||||
.await
|
||||
.map_err(admission_error)?;
|
||||
if !accepts_auth_namespace(
|
||||
&self.config,
|
||||
self.operator_allowlist.as_ref(),
|
||||
&expected.namespace,
|
||||
) {
|
||||
return Err(KubernetesDriverError::Precondition(
|
||||
"persisted runtime namespace is outside the driver's configured scope".into(),
|
||||
));
|
||||
}
|
||||
let lookup = self
|
||||
.supported_sandbox_api_for_lookup(self.client.clone())
|
||||
.await
|
||||
.map_err(KubernetesDriverError::Message)?;
|
||||
let objects = lookup
|
||||
.api
|
||||
.list(&ListParams::default().labels(&self.sandbox_lookup_selector(&request.sandbox_id)))
|
||||
.await
|
||||
.map_err(KubernetesDriverError::from_kube)?
|
||||
.items;
|
||||
let selected = select_expected_sandbox_runtime(objects, &expected)?;
|
||||
let name = selected.metadata.name.as_deref().ok_or_else(|| {
|
||||
KubernetesDriverError::Precondition("Sandbox resource has no name".into())
|
||||
})?;
|
||||
let api = Self::agent_sandbox_api(
|
||||
self.client.clone(),
|
||||
&lookup.resource.version,
|
||||
&expected.namespace,
|
||||
);
|
||||
// Refresh after selection, then use exactly this snapshot's version in
|
||||
// the patch. Never retry a resourceVersion conflict with a stale image,
|
||||
// attachment inventory or operating-state decision.
|
||||
let object = api
|
||||
.api
|
||||
.get(name)
|
||||
.await
|
||||
.map_err(KubernetesDriverError::from_kube)?;
|
||||
let object = select_expected_sandbox_runtime(vec![object], &expected)?;
|
||||
if sandbox_id_from_object(&object).as_deref() != Ok(request.sandbox_id.as_str())
|
||||
|| annotation_or_label(&object, LABEL_SANDBOX_NAME).as_deref()
|
||||
!= Some(request.name.as_str())
|
||||
{
|
||||
return Err(KubernetesDriverError::Precondition(
|
||||
"Sandbox resource does not match the requested sandbox identity".into(),
|
||||
));
|
||||
}
|
||||
require_stopped_for_image_update(&object, &lookup.resource.version)?;
|
||||
let mut candidate = object.clone();
|
||||
let repeat = prepare_update(&mut candidate, request, configuration)?;
|
||||
if let Some(configuration) = configuration {
|
||||
prepare_configuration(&mut candidate, configuration, repeat)?;
|
||||
}
|
||||
// Check the encoded candidate before preflight succeeds and the gateway
|
||||
// durably accepts an intent. Raw argv limits do not account for JSON
|
||||
// escaping, operation evidence, or existing annotations.
|
||||
let annotation_bytes: usize = candidate
|
||||
.metadata
|
||||
.annotations
|
||||
.iter()
|
||||
.flat_map(|annotations| annotations.iter())
|
||||
.map(|(key, value)| key.len() + value.len())
|
||||
.sum();
|
||||
if annotation_bytes > MAX_ANNOTATION_BYTES {
|
||||
return Err(KubernetesDriverError::InvalidArgument(format!(
|
||||
"sandbox update exceeds Kubernetes annotation size limit ({annotation_bytes} > {MAX_ANNOTATION_BYTES} bytes)"
|
||||
)));
|
||||
}
|
||||
// Revalidate the saved provenance and attachment UIDs against current
|
||||
// operator policy, even for retries. Do not manufacture new provenance
|
||||
// or re-render driver_config from a caller's request.
|
||||
self.admit_stored_resources(&candidate)
|
||||
.await
|
||||
.map_err(admission_error)?;
|
||||
self.require_image_update_pods_absent(&object, &request.sandbox_id, &expected.namespace)
|
||||
.await?;
|
||||
if request.validate_only || repeat {
|
||||
return Ok(());
|
||||
}
|
||||
let version = object
|
||||
.metadata
|
||||
.resource_version
|
||||
.as_deref()
|
||||
.filter(|version| !version.is_empty())
|
||||
.ok_or_else(|| {
|
||||
KubernetesDriverError::Precondition(
|
||||
"Sandbox resource has no resourceVersion".into(),
|
||||
)
|
||||
})?;
|
||||
let record = candidate
|
||||
.metadata
|
||||
.annotations
|
||||
.as_ref()
|
||||
.and_then(|annotations| annotations.get(IMAGE_UPDATE_ANNOTATION))
|
||||
.expect("prepared image update has operation evidence");
|
||||
let mut patch = serde_json::json!({
|
||||
"metadata": {
|
||||
"uid": expected.resource_uid,
|
||||
"resourceVersion": version,
|
||||
"annotations": {IMAGE_UPDATE_ANNOTATION: record}
|
||||
},
|
||||
"spec": {"podTemplate": {"spec": {
|
||||
"containers": candidate.data["spec"]["podTemplate"]["spec"]["containers"]
|
||||
}}}
|
||||
});
|
||||
if configuration.is_some_and(|configuration| configuration.main_process.is_some()) {
|
||||
let key = super::ANNOTATION_SANDBOX_RUNTIME_MAIN_PROCESS_SPEC;
|
||||
patch["metadata"]["annotations"][key] = serde_json::json!(
|
||||
candidate
|
||||
.metadata
|
||||
.annotations
|
||||
.as_ref()
|
||||
.and_then(|annotations| annotations.get(key))
|
||||
);
|
||||
}
|
||||
if let Some(init) = candidate
|
||||
.data
|
||||
.pointer("/spec/podTemplate/spec/initContainers")
|
||||
{
|
||||
patch["spec"]["podTemplate"]["spec"]["initContainers"] = init.clone();
|
||||
}
|
||||
api.api
|
||||
.patch(name, &PatchParams::default(), &Patch::Merge(&patch))
|
||||
.await
|
||||
.map_err(|error| {
|
||||
if is_kube_resource_version_conflict(&error) {
|
||||
KubernetesDriverError::Precondition(
|
||||
"Sandbox changed during image update; retry the same operation".into(),
|
||||
)
|
||||
} else {
|
||||
KubernetesDriverError::from_kube(error)
|
||||
}
|
||||
})?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn require_image_update_pods_absent(
|
||||
&self,
|
||||
object: &DynamicObject,
|
||||
sandbox_id: &str,
|
||||
namespace: &str,
|
||||
) -> Result<(), KubernetesDriverError> {
|
||||
let pods = Api::<Pod>::namespaced(self.client.clone(), namespace);
|
||||
let names = SandboxRuntimeNames::new(sandbox_id);
|
||||
// Check both the controller-recorded and default workload names. The
|
||||
// dedicated supervisor name is stable across generations; check it even
|
||||
// with shared supervision to catch a leftover dedicated Pod. Direct
|
||||
// reads keep preflight independent of the number of namespace Pods.
|
||||
let pod_names: BTreeSet<_> = [
|
||||
object.metadata.name.as_deref(),
|
||||
object
|
||||
.metadata
|
||||
.annotations
|
||||
.as_ref()
|
||||
.and_then(|annotations| annotations.get(SANDBOX_POD_NAME_ANNOTATION))
|
||||
.map(String::as_str),
|
||||
Some(names.supervisor_pod.as_str()),
|
||||
]
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.filter(|name| !name.is_empty())
|
||||
.collect();
|
||||
for name in pod_names {
|
||||
// Terminating Pods still block updates. Only a 404 proves absence;
|
||||
// authorization, transport and other API failures must propagate.
|
||||
if pods
|
||||
.get_opt(name)
|
||||
.await
|
||||
.map_err(KubernetesDriverError::from_kube)?
|
||||
.is_some()
|
||||
{
|
||||
return Err(KubernetesDriverError::Precondition(
|
||||
"cannot update image while sandbox workload or supervisor Pods still exist"
|
||||
.into(),
|
||||
));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
fn validate_image_update_request(
|
||||
request: &SandboxUpdatePreconditions,
|
||||
) -> Result<(), KubernetesDriverError> {
|
||||
for (field, value) in [
|
||||
("sandbox_id", request.sandbox_id.as_str()),
|
||||
("name", request.name.as_str()),
|
||||
("operation_id", request.operation_id.as_str()),
|
||||
] {
|
||||
if value.is_empty()
|
||||
|| value.len() > 1024
|
||||
|| value.trim() != value
|
||||
|| value.chars().any(char::is_control)
|
||||
{
|
||||
return Err(KubernetesDriverError::InvalidArgument(format!(
|
||||
"{field} must be a nonempty bounded identifier"
|
||||
)));
|
||||
}
|
||||
}
|
||||
if request.revision == 0 {
|
||||
return Err(KubernetesDriverError::InvalidArgument(
|
||||
"image update revision must be nonzero".into(),
|
||||
));
|
||||
}
|
||||
validate_workload_image(&request.image)?;
|
||||
// The old reference is only a compare-and-swap precondition, not a new
|
||||
// launch input. Legacy create accepted references that kubelet cannot pull;
|
||||
// validating their syntax here would prevent repairing those sandboxes.
|
||||
if request.expected_image.is_empty() || request.expected_image.len() > 2048 {
|
||||
return Err(KubernetesDriverError::InvalidArgument(
|
||||
"expected_image must be a nonempty bounded reference".into(),
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_workload_image(image: &str) -> Result<(), KubernetesDriverError> {
|
||||
// Distribution reference grammar: registry (including a port or bracketed
|
||||
// IPv6), lowercase repository path, optional tag and/or algorithm:digest.
|
||||
// Validation is syntax only: resolving/pulling the image remains kubelet's
|
||||
// job, using the existing operator-selected image-pull Secret inventory.
|
||||
static REFERENCE: LazyLock<regex::Regex> = LazyLock::new(|| {
|
||||
let host_component = r"[a-zA-Z0-9](?:[a-zA-Z0-9-]*[a-zA-Z0-9])?";
|
||||
let host = format!(r"(?:{host_component}(?:\.{host_component})*|\[[a-fA-F0-9:]+\])");
|
||||
let path = r"[a-z0-9]+(?:(?:[._]|__|-+)[a-z0-9]+)*";
|
||||
regex::Regex::new(&format!(
|
||||
r"^(?:{host}(?::[0-9]+)?/)?{path}(?:/{path})*(?::[a-zA-Z0-9_][a-zA-Z0-9_.-]{{0,127}})?(?:@[a-zA-Z][a-zA-Z0-9]*(?:[-_+.][a-zA-Z][a-zA-Z0-9]*)*:[a-fA-F0-9]{{32,}})?$"
|
||||
)).expect("valid distribution reference grammar")
|
||||
});
|
||||
let named = image.split('@').next().unwrap_or_default();
|
||||
let repository =
|
||||
named.rsplit_once(':').map_or(
|
||||
named,
|
||||
|(prefix, suffix)| {
|
||||
if suffix.contains('/') { named } else { prefix }
|
||||
},
|
||||
);
|
||||
let digest_length_valid = image.rsplit_once('@').is_none_or(|(_, digest)| {
|
||||
digest
|
||||
.split_once(':')
|
||||
.is_some_and(|(algorithm, encoded)| match algorithm {
|
||||
"sha256" => encoded.len() == 64,
|
||||
"sha384" => encoded.len() == 96,
|
||||
"sha512" => encoded.len() == 128,
|
||||
_ => true,
|
||||
})
|
||||
});
|
||||
if image.len() > 2048
|
||||
|| repository.len() > 255
|
||||
|| !digest_length_valid
|
||||
|| !REFERENCE.is_match(image)
|
||||
{
|
||||
return Err(KubernetesDriverError::InvalidArgument(
|
||||
"image must be a valid container image reference of at most 2048 bytes".into(),
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn require_stopped_for_image_update(
|
||||
object: &DynamicObject,
|
||||
version: &str,
|
||||
) -> Result<(), KubernetesDriverError> {
|
||||
let suspended = if version == SANDBOX_VERSION_V1BETA1 {
|
||||
object
|
||||
.data
|
||||
.pointer("/spec/operatingMode")
|
||||
.and_then(serde_json::Value::as_str)
|
||||
== Some("Suspended")
|
||||
} else {
|
||||
object
|
||||
.data
|
||||
.pointer("/spec/replicas")
|
||||
.and_then(serde_json::Value::as_i64)
|
||||
== Some(0)
|
||||
};
|
||||
if !suspended
|
||||
|| object.metadata.deletion_timestamp.is_some()
|
||||
|| sandbox_runtime_bootstrap_in_progress(object)
|
||||
{
|
||||
return Err(KubernetesDriverError::Precondition(
|
||||
"image updates require a stopped, suspended sandbox with no runtime bootstrap in progress".into(),
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Apply the image-only mutation in memory; return true for an exact retry.
|
||||
fn prepare_update(
|
||||
object: &mut DynamicObject,
|
||||
request: &SandboxUpdatePreconditions,
|
||||
configuration: Option<&openshell_core::proto::compute::v1::UpdateSandboxConfigurationRequest>,
|
||||
) -> Result<bool, KubernetesDriverError> {
|
||||
let evidence = object
|
||||
.metadata
|
||||
.annotations
|
||||
.as_ref()
|
||||
.and_then(|annotations| annotations.get(IMAGE_UPDATE_ANNOTATION))
|
||||
.map(|value| serde_json::from_str::<ImageUpdateRecord>(value))
|
||||
.transpose()
|
||||
.map_err(|_| {
|
||||
KubernetesDriverError::Precondition("invalid stored image update evidence".into())
|
||||
})?;
|
||||
let record = ImageUpdateRecord {
|
||||
operation_id: request.operation_id.clone(),
|
||||
expected_image: request.expected_image.clone(),
|
||||
image: request.image.clone(),
|
||||
revision: request.revision,
|
||||
command_updated: configuration.is_some_and(|value| value.main_process.is_some()),
|
||||
environment_updated: configuration.is_some_and(|value| value.environment.is_some()),
|
||||
};
|
||||
if let Some(saved) = &evidence {
|
||||
if request.revision < saved.revision {
|
||||
return Err(KubernetesDriverError::Precondition(
|
||||
"image update revision is older than the stored image revision".into(),
|
||||
));
|
||||
}
|
||||
if request.revision == saved.revision && saved != &record {
|
||||
return Err(KubernetesDriverError::Precondition(
|
||||
"image update revision was already used with different operation parameters".into(),
|
||||
));
|
||||
}
|
||||
if saved.operation_id == request.operation_id && saved != &record {
|
||||
return Err(KubernetesDriverError::Precondition(
|
||||
"image update operation_id was already used with different parameters".into(),
|
||||
));
|
||||
}
|
||||
}
|
||||
let repeat = evidence.as_ref() == Some(&record);
|
||||
let containers = object
|
||||
.data
|
||||
.pointer_mut("/spec/podTemplate/spec/containers")
|
||||
.and_then(serde_json::Value::as_array_mut)
|
||||
.ok_or_else(|| {
|
||||
KubernetesDriverError::Precondition("Sandbox has no workload containers".into())
|
||||
})?;
|
||||
let mut agents = containers
|
||||
.iter_mut()
|
||||
.filter(|container| container["name"] == "agent");
|
||||
let agent = agents.next().ok_or_else(|| {
|
||||
KubernetesDriverError::Precondition("Sandbox has no agent container".into())
|
||||
})?;
|
||||
if agents.next().is_some() {
|
||||
return Err(KubernetesDriverError::Precondition(
|
||||
"Sandbox has multiple agent containers".into(),
|
||||
));
|
||||
}
|
||||
let expected: &str = if repeat {
|
||||
&request.image
|
||||
} else {
|
||||
&request.expected_image
|
||||
};
|
||||
if agent["image"].as_str() != Some(expected) {
|
||||
return Err(KubernetesDriverError::Precondition(
|
||||
"Sandbox workload image does not match expected_image or the recorded operation target"
|
||||
.into(),
|
||||
));
|
||||
}
|
||||
agent["image"] = serde_json::json!(request.image);
|
||||
if let Some(init) = object
|
||||
.data
|
||||
.pointer_mut("/spec/podTemplate/spec/initContainers")
|
||||
.and_then(serde_json::Value::as_array_mut)
|
||||
{
|
||||
for container in init
|
||||
.iter_mut()
|
||||
.filter(|container| container["name"] == WORKSPACE_INIT_CONTAINER_NAME)
|
||||
{
|
||||
if container["image"].as_str() != Some(expected) {
|
||||
return Err(KubernetesDriverError::Precondition(
|
||||
"Sandbox workspace seed image does not match its workload image".into(),
|
||||
));
|
||||
}
|
||||
container["image"] = serde_json::json!(request.image);
|
||||
}
|
||||
}
|
||||
object
|
||||
.metadata
|
||||
.annotations
|
||||
.get_or_insert_with(BTreeMap::new)
|
||||
.insert(
|
||||
IMAGE_UPDATE_ANNOTATION.into(),
|
||||
serde_json::to_string(&record).expect("image update record serializes"),
|
||||
);
|
||||
Ok(repeat)
|
||||
}
|
||||
|
||||
fn validate_configuration(
|
||||
request: &openshell_core::proto::compute::v1::UpdateSandboxConfigurationRequest,
|
||||
) -> Result<(), KubernetesDriverError> {
|
||||
let invalid = |message: &str| KubernetesDriverError::InvalidArgument(message.into());
|
||||
if let Some(spec) = &request.main_process
|
||||
&& (spec.command.len() > 256
|
||||
|| spec.command.iter().map(String::len).sum::<usize>() > 256 * 1024
|
||||
|| spec.command.first().is_some_and(String::is_empty)
|
||||
|| spec.command.iter().any(|arg| arg.contains('\0')))
|
||||
{
|
||||
return Err(invalid("invalid workload argv"));
|
||||
}
|
||||
if let Some(environment) = &request.environment {
|
||||
if environment.values.len() > 256
|
||||
|| environment
|
||||
.values
|
||||
.iter()
|
||||
.map(|(key, value)| key.len() + value.len())
|
||||
.sum::<usize>()
|
||||
> 256 * 1024
|
||||
{
|
||||
return Err(invalid("environment exceeds size limits"));
|
||||
}
|
||||
for (key, value) in &environment.values {
|
||||
if key.is_empty()
|
||||
|| key.starts_with("OPENSHELL_")
|
||||
|| key.as_bytes()[0].is_ascii_digit()
|
||||
|| !key.bytes().all(|c| c.is_ascii_alphanumeric() || c == b'_')
|
||||
|| value.contains(['\0', '\n', '\r'])
|
||||
{
|
||||
return Err(invalid("invalid or reserved workload environment entry"));
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn prepare_configuration(
|
||||
object: &mut DynamicObject,
|
||||
request: &openshell_core::proto::compute::v1::UpdateSandboxConfigurationRequest,
|
||||
repeat: bool,
|
||||
) -> Result<(), KubernetesDriverError> {
|
||||
if let Some(spec) = &request.main_process {
|
||||
let key = super::ANNOTATION_SANDBOX_RUNTIME_MAIN_PROCESS_SPEC;
|
||||
let encoded = super::required_sandbox_annotation(object, key)?;
|
||||
let mut main = openshell_core::sandbox_env::MainProcessConfig::decode(&encoded)
|
||||
.map_err(KubernetesDriverError::Precondition)?;
|
||||
let original = main.clone();
|
||||
main.command.clone_from(&spec.command);
|
||||
main.tty = spec.command.is_empty() || spec.tty;
|
||||
if repeat && main != original {
|
||||
return Err(KubernetesDriverError::Precondition(
|
||||
"recorded update command does not match target".into(),
|
||||
));
|
||||
}
|
||||
object
|
||||
.metadata
|
||||
.annotations
|
||||
.get_or_insert_with(BTreeMap::new)
|
||||
.insert(
|
||||
key.into(),
|
||||
serde_json::to_string(&main).map_err(|_| {
|
||||
KubernetesDriverError::InvalidArgument("cannot encode main process".into())
|
||||
})?,
|
||||
);
|
||||
}
|
||||
if let Some(environment) = &request.environment {
|
||||
let containers = object
|
||||
.data
|
||||
.pointer_mut("/spec/podTemplate/spec/containers")
|
||||
.and_then(serde_json::Value::as_array_mut)
|
||||
.ok_or_else(|| {
|
||||
KubernetesDriverError::Precondition("missing workload containers".into())
|
||||
})?;
|
||||
let agent = containers
|
||||
.iter_mut()
|
||||
.find(|container| container["name"] == "agent")
|
||||
.ok_or_else(|| KubernetesDriverError::Precondition("missing agent container".into()))?;
|
||||
let mut current = std::collections::HashMap::new();
|
||||
let mut retained = Vec::new();
|
||||
for entry in agent["env"].as_array().into_iter().flatten() {
|
||||
let name = entry["name"].as_str().ok_or_else(|| {
|
||||
KubernetesDriverError::Precondition("invalid workload environment".into())
|
||||
})?;
|
||||
if name.starts_with("OPENSHELL_") {
|
||||
retained.push(entry.clone());
|
||||
} else if let Some(value) = entry["value"].as_str() {
|
||||
current.insert(name.to_owned(), value.to_owned());
|
||||
} else {
|
||||
return Err(KubernetesDriverError::Precondition(
|
||||
"cannot replace workload environment with external value sources".into(),
|
||||
));
|
||||
}
|
||||
}
|
||||
if repeat && current != environment.values {
|
||||
return Err(KubernetesDriverError::Precondition(
|
||||
"recorded update environment does not match target".into(),
|
||||
));
|
||||
}
|
||||
let mut entries: Vec<_> = environment.values.iter().collect();
|
||||
entries.sort_unstable_by_key(|(name, _)| *name);
|
||||
for (name, value) in entries {
|
||||
retained.push(serde_json::json!({"name": name, "value": value}));
|
||||
}
|
||||
agent["env"] = serde_json::json!(retained);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "image_update_tests.rs"]
|
||||
mod tests;
|
||||
File diff suppressed because it is too large
Load Diff
@@ -339,7 +339,9 @@ async fn main() -> Result<()> {
|
||||
.await
|
||||
.into_diagnostic()?;
|
||||
|
||||
let service = ComputeDriverServer::new(ComputeDriverService::new(driver));
|
||||
let driver_service = ComputeDriverService::new(driver);
|
||||
let configuration_updates_service = openshell_core::proto::compute::v1::compute_driver_configuration_updates_server::ComputeDriverConfigurationUpdatesServer::new(driver_service.clone());
|
||||
let service = ComputeDriverServer::new(driver_service);
|
||||
let shutdown = async move {
|
||||
shutdown_signal().await;
|
||||
let _ = shutdown_tx.send(true);
|
||||
@@ -353,6 +355,7 @@ async fn main() -> Result<()> {
|
||||
tonic::transport::Server::builder()
|
||||
.layer(openshell_otel::compute_driver_rpc_layer())
|
||||
.add_service(service)
|
||||
.add_service(configuration_updates_service)
|
||||
.serve_with_incoming_shutdown(
|
||||
openshell_core::external_driver_socket::SameUidUnixIncoming::new(listener),
|
||||
shutdown,
|
||||
@@ -364,6 +367,7 @@ async fn main() -> Result<()> {
|
||||
tonic::transport::Server::builder()
|
||||
.layer(openshell_otel::compute_driver_rpc_layer())
|
||||
.add_service(service)
|
||||
.add_service(configuration_updates_service)
|
||||
.serve_with_shutdown(args.bind_address, shutdown)
|
||||
.await
|
||||
.into_diagnostic()
|
||||
|
||||
@@ -32,6 +32,9 @@ impl ComputeDriverService {
|
||||
}
|
||||
}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl openshell_core::compute::ComputeDriver for ComputeDriverService {}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl ComputeDriver for ComputeDriverService {
|
||||
async fn get_capabilities(
|
||||
|
||||
@@ -48,6 +48,9 @@ impl ComputeDriverService {
|
||||
}
|
||||
}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl openshell_core::compute::ComputeDriver for ComputeDriverService {}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl ComputeDriver for ComputeDriverService {
|
||||
async fn authenticate_sandbox(
|
||||
|
||||
@@ -5323,6 +5323,9 @@ fn configure_main_exit_marker(command: &mut Command, state_dir: &Path) {
|
||||
.arg(state_dir.join(MAIN_PROCESS_EXITED_FILE));
|
||||
}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl openshell_core::compute::ComputeDriver for VmDriver {}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl ComputeDriver for VmDriver {
|
||||
async fn authenticate_sandbox(
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
//! Interceptable `OpenShell` route classification.
|
||||
//! Interceptable public gateway route classification.
|
||||
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
|
||||
@@ -10,6 +10,10 @@ use prost_reflect::DescriptorPool;
|
||||
use crate::{InterceptorError, Result};
|
||||
|
||||
const SERVICE_OPEN_SHELL: &str = "openshell.v1.OpenShell";
|
||||
const SERVICE_SANDBOX_UPDATES: &str = "openshell.v1.SandboxUpdates";
|
||||
|
||||
/// Unary sandbox-update methods that may be targeted by gateway interceptors.
|
||||
pub const INTERCEPTABLE_SANDBOX_UPDATE_METHODS: &[&str] = &["UpdateSandbox"];
|
||||
|
||||
/// Unary `openshell.v1.OpenShell` methods that may be targeted by gateway
|
||||
/// interceptors. New methods are non-interceptable until deliberately added
|
||||
@@ -58,25 +62,31 @@ impl OpenShellRouteIndex {
|
||||
}
|
||||
|
||||
pub(crate) fn from_descriptor_pool(pool: &DescriptorPool) -> Result<Self> {
|
||||
let service = pool
|
||||
let open_shell = pool
|
||||
.get_service_by_name(SERVICE_OPEN_SHELL)
|
||||
.ok_or_else(|| {
|
||||
InterceptorError::Config(format!(
|
||||
"descriptor set does not contain service '{SERVICE_OPEN_SHELL}'"
|
||||
))
|
||||
})?;
|
||||
// SandboxUpdates is an optional extension: baseline descriptor sets
|
||||
// still need only OpenShell, but a present extension must be valid.
|
||||
let sandbox_updates = pool.get_service_by_name(SERVICE_SANDBOX_UPDATES);
|
||||
let mut all_methods = BTreeSet::new();
|
||||
let mut unary_methods = BTreeSet::new();
|
||||
let mut input_types = BTreeMap::new();
|
||||
let mut output_types = BTreeMap::new();
|
||||
|
||||
for method in service.methods() {
|
||||
let name = method.name().to_string();
|
||||
all_methods.insert(name.clone());
|
||||
if !method.is_client_streaming() && !method.is_server_streaming() {
|
||||
unary_methods.insert(name.clone());
|
||||
input_types.insert(name.clone(), method.input().full_name().to_string());
|
||||
output_types.insert(name, method.output().full_name().to_string());
|
||||
for service in std::iter::once(&open_shell).chain(sandbox_updates.as_ref()) {
|
||||
let service_name = service.full_name();
|
||||
for method in service.methods() {
|
||||
let route = format!("{service_name}/{}", method.name());
|
||||
all_methods.insert(route.clone());
|
||||
if !method.is_client_streaming() && !method.is_server_streaming() {
|
||||
unary_methods.insert(route.clone());
|
||||
input_types.insert(route.clone(), method.input().full_name().to_string());
|
||||
output_types.insert(route, method.output().full_name().to_string());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -86,43 +96,49 @@ impl OpenShellRouteIndex {
|
||||
input_types,
|
||||
output_types,
|
||||
};
|
||||
index.validate_interceptable_list()?;
|
||||
index.validate_interceptable_list(SERVICE_OPEN_SHELL, INTERCEPTABLE_METHODS)?;
|
||||
if sandbox_updates.is_some() {
|
||||
index.validate_interceptable_list(
|
||||
SERVICE_SANDBOX_UPDATES,
|
||||
INTERCEPTABLE_SANDBOX_UPDATE_METHODS,
|
||||
)?;
|
||||
}
|
||||
Ok(index)
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn is_interceptable(&self, service: &str, method: &str) -> bool {
|
||||
service == SERVICE_OPEN_SHELL
|
||||
&& self.unary_methods.contains(method)
|
||||
&& INTERCEPTABLE_METHODS.contains(&method)
|
||||
let allowed = match service {
|
||||
SERVICE_OPEN_SHELL => INTERCEPTABLE_METHODS,
|
||||
SERVICE_SANDBOX_UPDATES => INTERCEPTABLE_SANDBOX_UPDATE_METHODS,
|
||||
_ => return false,
|
||||
};
|
||||
allowed.contains(&method) && self.unary_methods.contains(&format!("{service}/{method}"))
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn input_type(&self, service: &str, method: &str) -> Option<&str> {
|
||||
if service == SERVICE_OPEN_SHELL && self.unary_methods.contains(method) {
|
||||
self.input_types.get(method).map(String::as_str)
|
||||
} else {
|
||||
None
|
||||
}
|
||||
self.input_types
|
||||
.get(&format!("{service}/{method}"))
|
||||
.map(String::as_str)
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn output_type(&self, service: &str, method: &str) -> Option<&str> {
|
||||
if service == SERVICE_OPEN_SHELL && self.unary_methods.contains(method) {
|
||||
self.output_types.get(method).map(String::as_str)
|
||||
} else {
|
||||
None
|
||||
}
|
||||
self.output_types
|
||||
.get(&format!("{service}/{method}"))
|
||||
.map(String::as_str)
|
||||
}
|
||||
|
||||
fn validate_interceptable_list(&self) -> Result<()> {
|
||||
fn validate_interceptable_list(&self, service: &str, methods: &[&str]) -> Result<()> {
|
||||
let mut stale = Vec::new();
|
||||
let mut streaming = Vec::new();
|
||||
for method in INTERCEPTABLE_METHODS {
|
||||
if !self.all_methods.contains(*method) {
|
||||
stale.push((*method).to_string());
|
||||
} else if !self.unary_methods.contains(*method) {
|
||||
streaming.push((*method).to_string());
|
||||
for method in methods {
|
||||
let route = format!("{service}/{method}");
|
||||
if !self.all_methods.contains(&route) {
|
||||
stale.push(route);
|
||||
} else if !self.unary_methods.contains(&route) {
|
||||
streaming.push(route);
|
||||
}
|
||||
}
|
||||
if !stale.is_empty() {
|
||||
@@ -141,8 +157,158 @@ impl OpenShellRouteIndex {
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use prost::Message;
|
||||
use prost_types::{FileDescriptorSet, ServiceDescriptorProto};
|
||||
|
||||
use super::*;
|
||||
|
||||
fn descriptor_set() -> FileDescriptorSet {
|
||||
FileDescriptorSet::decode(openshell_core::FILE_DESCRIPTOR_SET).unwrap()
|
||||
}
|
||||
|
||||
fn service_mut<'a>(
|
||||
descriptors: &'a mut FileDescriptorSet,
|
||||
name: &str,
|
||||
) -> &'a mut ServiceDescriptorProto {
|
||||
descriptors
|
||||
.file
|
||||
.iter_mut()
|
||||
.filter(|file| file.package.as_deref() == Some("openshell.v1"))
|
||||
.flat_map(|file| &mut file.service)
|
||||
.find(|service| service.name.as_deref() == Some(name))
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn open_shell_only_descriptor_set_is_accepted() {
|
||||
let mut descriptors = descriptor_set();
|
||||
for file in &mut descriptors.file {
|
||||
file.service.retain(|service| {
|
||||
file.package.as_deref() == Some("openshell.v1")
|
||||
&& service.name.as_deref() == Some("OpenShell")
|
||||
});
|
||||
}
|
||||
|
||||
let index = OpenShellRouteIndex::from_descriptor_set(&descriptors.encode_to_vec()).unwrap();
|
||||
for method in INTERCEPTABLE_METHODS {
|
||||
assert!(index.is_interceptable(SERVICE_OPEN_SHELL, method));
|
||||
assert!(index.input_type(SERVICE_OPEN_SHELL, method).is_some());
|
||||
assert!(index.output_type(SERVICE_OPEN_SHELL, method).is_some());
|
||||
}
|
||||
assert!(!index.is_interceptable(SERVICE_SANDBOX_UPDATES, "UpdateSandbox"));
|
||||
assert!(
|
||||
index
|
||||
.input_type(SERVICE_SANDBOX_UPDATES, "UpdateSandbox")
|
||||
.is_none()
|
||||
);
|
||||
assert!(
|
||||
index
|
||||
.output_type(SERVICE_SANDBOX_UPDATES, "UpdateSandbox")
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn missing_open_shell_service_is_rejected() {
|
||||
let mut descriptors = descriptor_set();
|
||||
for file in &mut descriptors.file {
|
||||
file.service
|
||||
.retain(|service| service.name.as_deref() != Some("OpenShell"));
|
||||
}
|
||||
|
||||
let error =
|
||||
OpenShellRouteIndex::from_descriptor_set(&descriptors.encode_to_vec()).unwrap_err();
|
||||
assert!(error.to_string().contains(&format!(
|
||||
"descriptor set does not contain service '{SERVICE_OPEN_SHELL}'"
|
||||
)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn present_sandbox_updates_service_requires_update_sandbox() {
|
||||
let mut descriptors = descriptor_set();
|
||||
service_mut(&mut descriptors, "SandboxUpdates")
|
||||
.method
|
||||
.clear();
|
||||
|
||||
let error =
|
||||
OpenShellRouteIndex::from_descriptor_set(&descriptors.encode_to_vec()).unwrap_err();
|
||||
assert!(error.to_string().contains("stale methods"));
|
||||
assert!(
|
||||
error
|
||||
.to_string()
|
||||
.contains("openshell.v1.SandboxUpdates/UpdateSandbox")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn streaming_update_sandbox_is_rejected() {
|
||||
for (client_streaming, server_streaming) in [(true, false), (false, true), (true, true)] {
|
||||
let mut descriptors = descriptor_set();
|
||||
let method = &mut service_mut(&mut descriptors, "SandboxUpdates").method[0];
|
||||
assert_eq!(method.name.as_deref(), Some("UpdateSandbox"));
|
||||
method.client_streaming = Some(client_streaming);
|
||||
method.server_streaming = Some(server_streaming);
|
||||
|
||||
let error =
|
||||
OpenShellRouteIndex::from_descriptor_set(&descriptors.encode_to_vec()).unwrap_err();
|
||||
assert!(error.to_string().contains("streaming methods"));
|
||||
assert!(
|
||||
error
|
||||
.to_string()
|
||||
.contains("openshell.v1.SandboxUpdates/UpdateSandbox")
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unlisted_sandbox_update_methods_are_not_interceptable() {
|
||||
let mut descriptors = descriptor_set();
|
||||
let service = service_mut(&mut descriptors, "SandboxUpdates");
|
||||
let mut future_method = service.method[0].clone();
|
||||
future_method.name = Some("FutureUnaryMethod".to_string());
|
||||
service.method.push(future_method);
|
||||
|
||||
let index = OpenShellRouteIndex::from_descriptor_set(&descriptors.encode_to_vec()).unwrap();
|
||||
assert!(index.is_interceptable(SERVICE_SANDBOX_UPDATES, "UpdateSandbox"));
|
||||
assert!(!index.is_interceptable(SERVICE_SANDBOX_UPDATES, "FutureUnaryMethod"));
|
||||
assert_eq!(
|
||||
index.input_type(SERVICE_SANDBOX_UPDATES, "FutureUnaryMethod"),
|
||||
Some("openshell.v1.UpdateSandboxRequest")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn open_shell_allowlist_validation_remains_required() {
|
||||
let mut missing_method = descriptor_set();
|
||||
service_mut(&mut missing_method, "OpenShell")
|
||||
.method
|
||||
.retain(|method| method.name.as_deref() != Some("CreateSandbox"));
|
||||
let error =
|
||||
OpenShellRouteIndex::from_descriptor_set(&missing_method.encode_to_vec()).unwrap_err();
|
||||
assert!(error.to_string().contains("stale methods"));
|
||||
assert!(
|
||||
error
|
||||
.to_string()
|
||||
.contains("openshell.v1.OpenShell/CreateSandbox")
|
||||
);
|
||||
|
||||
let mut streaming_method = descriptor_set();
|
||||
let method = service_mut(&mut streaming_method, "OpenShell")
|
||||
.method
|
||||
.iter_mut()
|
||||
.find(|method| method.name.as_deref() == Some("CreateSandbox"))
|
||||
.unwrap();
|
||||
method.server_streaming = Some(true);
|
||||
let error = OpenShellRouteIndex::from_descriptor_set(&streaming_method.encode_to_vec())
|
||||
.unwrap_err();
|
||||
assert!(error.to_string().contains("streaming methods"));
|
||||
assert!(
|
||||
error
|
||||
.to_string()
|
||||
.contains("openshell.v1.OpenShell/CreateSandbox")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn interceptable_entries_match_real_unary_methods() {
|
||||
OpenShellRouteIndex::from_descriptor_set(openshell_core::FILE_DESCRIPTOR_SET).unwrap();
|
||||
@@ -153,6 +319,26 @@ mod tests {
|
||||
let index =
|
||||
OpenShellRouteIndex::from_descriptor_set(openshell_core::FILE_DESCRIPTOR_SET).unwrap();
|
||||
assert!(index.is_interceptable("openshell.v1.OpenShell", "CreateSandbox"));
|
||||
assert!(index.is_interceptable(SERVICE_SANDBOX_UPDATES, "UpdateSandbox"));
|
||||
assert!(!index.is_interceptable(SERVICE_OPEN_SHELL, "UpdateSandbox"));
|
||||
assert!(!index.is_interceptable(SERVICE_SANDBOX_UPDATES, "CreateSandbox"));
|
||||
assert_eq!(
|
||||
index.input_type(SERVICE_OPEN_SHELL, "CreateSandbox"),
|
||||
Some("openshell.v1.CreateSandboxRequest")
|
||||
);
|
||||
assert_eq!(
|
||||
index.input_type(SERVICE_SANDBOX_UPDATES, "UpdateSandbox"),
|
||||
Some("openshell.v1.UpdateSandboxRequest")
|
||||
);
|
||||
assert_eq!(
|
||||
index.output_type(SERVICE_SANDBOX_UPDATES, "UpdateSandbox"),
|
||||
Some("openshell.v1.SandboxResponse")
|
||||
);
|
||||
assert!(
|
||||
index
|
||||
.input_type(SERVICE_OPEN_SHELL, "UpdateSandbox")
|
||||
.is_none()
|
||||
);
|
||||
assert!(index.is_interceptable("openshell.v1.OpenShell", "UpdateConfig"));
|
||||
assert!(index.is_interceptable("openshell.v1.OpenShell", "SubmitPolicyAnalysis"));
|
||||
assert!(!index.is_interceptable("openshell.v1.OpenShell", "Health"));
|
||||
|
||||
@@ -286,6 +286,18 @@ impl openshell_server::ComputeDriverFactory for KubernetesFactory {
|
||||
&self,
|
||||
context: openshell_server::ComputeDriverBuildContext<'_>,
|
||||
) -> openshell_core::Result<openshell_server::ComputeDriverInstance> {
|
||||
self.build_with_configuration_updates(context)
|
||||
.await
|
||||
.map(|(instance, _)| instance)
|
||||
}
|
||||
|
||||
async fn build_with_configuration_updates(
|
||||
&self,
|
||||
context: openshell_server::ComputeDriverBuildContext<'_>,
|
||||
) -> openshell_core::Result<(
|
||||
openshell_server::ComputeDriverInstance,
|
||||
Option<openshell_server::SharedComputeDriverConfigurationUpdates>,
|
||||
)> {
|
||||
let mut config = kubernetes_config(context.config_context())?;
|
||||
config.supervisor_otlp_endpoint = context
|
||||
.otlp_config()
|
||||
@@ -327,9 +339,12 @@ impl openshell_server::ComputeDriverFactory for KubernetesFactory {
|
||||
.await
|
||||
}
|
||||
.map_err(|error| openshell_core::Error::execution(error.to_string()))?;
|
||||
let driver = openshell_driver_kubernetes::ComputeDriverService::new_in_process(driver);
|
||||
Ok(openshell_server::ComputeDriverInstance::InProcess(
|
||||
std::sync::Arc::new(driver),
|
||||
let driver = std::sync::Arc::new(
|
||||
openshell_driver_kubernetes::ComputeDriverService::new_in_process(driver),
|
||||
);
|
||||
Ok((
|
||||
openshell_server::ComputeDriverInstance::InProcess(driver.clone()),
|
||||
Some(driver),
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -102,6 +102,11 @@ pub mod rpc {
|
||||
"CreateSandbox",
|
||||
"openshell.compute.v1.ComputeDriver/CreateSandbox",
|
||||
);
|
||||
pub const UPDATE_SANDBOX_CONFIGURATION: ComputeDriverRpc = ComputeDriverRpc {
|
||||
service: "openshell.compute.v1.ComputeDriverConfigurationUpdates",
|
||||
method: "UpdateSandboxConfiguration",
|
||||
operation: "openshell.compute.v1.ComputeDriverConfigurationUpdates/UpdateSandboxConfiguration",
|
||||
};
|
||||
pub const GET_SANDBOX: ComputeDriverRpc = ComputeDriverRpc::new(
|
||||
"GetSandbox",
|
||||
"openshell.compute.v1.ComputeDriver/GetSandbox",
|
||||
@@ -194,6 +199,7 @@ pub fn compute_driver_rpc_operation(path: &str) -> Option<ComputeDriverRpc> {
|
||||
Some("GetCapabilities") => Some(rpc::GET_CAPABILITIES),
|
||||
Some("ValidateSandboxCreate") => Some(rpc::VALIDATE_SANDBOX_CREATE),
|
||||
Some("CreateSandbox") => Some(rpc::CREATE_SANDBOX),
|
||||
Some("UpdateSandboxConfiguration") => Some(rpc::UPDATE_SANDBOX_CONFIGURATION),
|
||||
Some("GetSandbox") => Some(rpc::GET_SANDBOX),
|
||||
Some("ListSandboxes") => Some(rpc::LIST_SANDBOXES),
|
||||
Some("StopSandbox") => Some(rpc::STOP_SANDBOX),
|
||||
@@ -317,6 +323,7 @@ mod tests {
|
||||
rpc::GET_CAPABILITIES,
|
||||
rpc::VALIDATE_SANDBOX_CREATE,
|
||||
rpc::CREATE_SANDBOX,
|
||||
rpc::UPDATE_SANDBOX_CONFIGURATION,
|
||||
rpc::GET_SANDBOX,
|
||||
rpc::LIST_SANDBOXES,
|
||||
rpc::STOP_SANDBOX,
|
||||
|
||||
@@ -12,7 +12,7 @@ use crate::auth::{BearerSlot, EdgeAuthInterceptor, bearer_metadata};
|
||||
use crate::config::{AuthConfig, ClientConfig};
|
||||
use crate::error::{Result, SdkError};
|
||||
use crate::pagination::{Page, Pager};
|
||||
use crate::raw::AuthedGrpcClient;
|
||||
use crate::raw::{AuthedGrpcClient, AuthedSandboxUpdatesClient};
|
||||
use crate::refresh::{RefreshedToken, TokenSource};
|
||||
use crate::routing::OwnerCache;
|
||||
use crate::types::{
|
||||
@@ -122,6 +122,23 @@ impl OpenShellClient {
|
||||
Ok(self.raw_grpc())
|
||||
}
|
||||
|
||||
/// Authenticated client for the optional `SandboxUpdates` service, sharing
|
||||
/// this client's channel and live bearer slot. Like [`Self::raw_grpc`], this
|
||||
/// does not refresh or retry OIDC credentials by itself.
|
||||
pub fn raw_sandbox_updates(&self) -> AuthedSandboxUpdatesClient {
|
||||
proto::sandbox_updates_client::SandboxUpdatesClient::with_interceptor(
|
||||
self.channel.clone(),
|
||||
self.interceptor.clone(),
|
||||
)
|
||||
}
|
||||
|
||||
/// Like [`Self::raw_sandbox_updates`], but refreshes the bearer token first
|
||||
/// when needed. Reactive retry remains the raw caller's responsibility.
|
||||
pub async fn raw_sandbox_updates_fresh(&self) -> Result<AuthedSandboxUpdatesClient> {
|
||||
self.ensure_fresh().await?;
|
||||
Ok(self.raw_sandbox_updates())
|
||||
}
|
||||
|
||||
/// Force an OIDC refresh and write the new token into the live bearer
|
||||
/// slot, regardless of expiry. Returns `true` when a refresher is wired
|
||||
/// and a fresh token was minted, `false` for static auth. Use after a
|
||||
|
||||
@@ -20,6 +20,7 @@
|
||||
|
||||
pub use openshell_core::proto;
|
||||
pub use openshell_core::proto::open_shell_client::OpenShellClient as GrpcClient;
|
||||
pub use openshell_core::proto::sandbox_updates_client::SandboxUpdatesClient;
|
||||
pub use openshell_core::proto::{
|
||||
CreateSandboxRequest, CreateSandboxTemplateRequest, CreateWorkspaceRequest,
|
||||
DeleteSandboxRequest, DeleteSandboxTemplateRequest, DeleteWorkspaceRequest, ExecSandboxRequest,
|
||||
@@ -28,12 +29,12 @@ pub use openshell_core::proto::{
|
||||
ListSandboxTemplatesRequest, ListSandboxesRequest, ListWorkspacesRequest,
|
||||
ProviderDesiredIdentity, ProviderMutationKind, ProviderMutationReceipt,
|
||||
ProviderReadinessObservation, ProviderReadinessReason, ProviderReadinessState,
|
||||
ProviderReadinessStatus, ResumeSandboxRequest, Sandbox, SandboxPhase as ProtoSandboxPhase,
|
||||
SandboxResources, SandboxServiceLevel, SandboxSpec as ProtoSandboxSpec, SandboxStartup,
|
||||
SandboxTemplate, SandboxTemplateResponse, SandboxWorkloadConfig, SandboxWorkloadTemplate,
|
||||
SandboxWorkloadTemplateProvenance, SandboxWorkloadTemplateSpec,
|
||||
ServiceStatus as ProtoServiceStatus, StartSandboxRequest, StopSandboxRequest,
|
||||
SuspendSandboxRequest, Workspace,
|
||||
ProviderReadinessStatus, ResumeSandboxRequest, Sandbox, SandboxCommandUpdate, SandboxMapUpdate,
|
||||
SandboxPhase as ProtoSandboxPhase, SandboxResources, SandboxServiceLevel,
|
||||
SandboxSpec as ProtoSandboxSpec, SandboxStartup, SandboxTemplate, SandboxTemplateResponse,
|
||||
SandboxWorkloadConfig, SandboxWorkloadTemplate, SandboxWorkloadTemplateProvenance,
|
||||
SandboxWorkloadTemplateSpec, ServiceStatus as ProtoServiceStatus, StartSandboxRequest,
|
||||
StopSandboxRequest, SuspendSandboxRequest, UpdateSandboxRequest, Workspace,
|
||||
};
|
||||
|
||||
/// Type alias for the gRPC client wrapped in the SDK's auth interceptor.
|
||||
@@ -43,3 +44,12 @@ pub type AuthedGrpcClient = GrpcClient<
|
||||
crate::EdgeAuthInterceptor,
|
||||
>,
|
||||
>;
|
||||
|
||||
/// Sandbox-updates client using the same live auth slot and transport as the
|
||||
/// main service client.
|
||||
pub type AuthedSandboxUpdatesClient = SandboxUpdatesClient<
|
||||
tonic::service::interceptor::InterceptedService<
|
||||
tonic::transport::Channel,
|
||||
crate::EdgeAuthInterceptor,
|
||||
>,
|
||||
>;
|
||||
|
||||
@@ -10,6 +10,7 @@
|
||||
|
||||
use openshell_core::proto;
|
||||
use openshell_core::proto::open_shell_server::{OpenShell, OpenShellServer};
|
||||
use openshell_core::proto::sandbox_updates_server::{SandboxUpdates, SandboxUpdatesServer};
|
||||
use openshell_core::replica_routing;
|
||||
use openshell_sdk::{
|
||||
AuthConfig, ClientConfig, ExecOptions, ListOptions, OpenShellClient, Refresh, RefreshError,
|
||||
@@ -60,6 +61,8 @@ struct MockState {
|
||||
last_start: Mutex<Option<proto::StartSandboxRequest>>,
|
||||
last_suspend: Mutex<Option<proto::SuspendSandboxRequest>>,
|
||||
last_resume: Mutex<Option<proto::ResumeSandboxRequest>>,
|
||||
|
||||
last_update: Mutex<Option<proto::UpdateSandboxRequest>>,
|
||||
last_list_request: Mutex<Option<proto::ListSandboxesRequest>>,
|
||||
list_requests: Mutex<Vec<proto::ListSandboxesRequest>>,
|
||||
last_exec_request: Mutex<Option<proto::ExecSandboxRequest>>,
|
||||
@@ -1172,16 +1175,40 @@ impl OpenShell for TestOpenShell {
|
||||
}
|
||||
}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl SandboxUpdates for TestOpenShell {
|
||||
async fn update_sandbox(
|
||||
&self,
|
||||
request: tonic::Request<proto::UpdateSandboxRequest>,
|
||||
) -> Result<Response<proto::SandboxResponse>, Status> {
|
||||
if let Some(expected) = &self.state.require_bearer {
|
||||
let got = request
|
||||
.metadata()
|
||||
.get("authorization")
|
||||
.and_then(|value| value.to_str().ok());
|
||||
if got != Some(expected.as_str()) {
|
||||
return Err(Status::unauthenticated("invalid or missing bearer token"));
|
||||
}
|
||||
}
|
||||
*self.state.last_update.lock().await = Some(request.into_inner());
|
||||
Ok(Response::new(proto::SandboxResponse::default()))
|
||||
}
|
||||
}
|
||||
|
||||
/// Spin up the mock gateway, return its endpoint URL.
|
||||
async fn start_mock(state: Arc<MockState>) -> String {
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let addr = listener.local_addr().unwrap();
|
||||
let endpoint = format!("http://{addr}");
|
||||
let stream = TcpListenerStream::new(listener);
|
||||
let updates = SandboxUpdatesServer::new(TestOpenShell {
|
||||
state: state.clone(),
|
||||
});
|
||||
let svc = OpenShellServer::new(TestOpenShell { state });
|
||||
tokio::spawn(async move {
|
||||
let _ = tonic::transport::Server::builder()
|
||||
.add_service(svc)
|
||||
.add_service(updates)
|
||||
.serve_with_incoming(stream)
|
||||
.await;
|
||||
});
|
||||
@@ -2371,6 +2398,65 @@ async fn raw_grpc_fresh_refreshes_before_raw_call() {
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn raw_sandbox_updates_shares_transport_and_live_auth_slot() {
|
||||
let state = Arc::new(MockState {
|
||||
require_bearer: Some("Bearer fresh-token".to_string()),
|
||||
..Default::default()
|
||||
});
|
||||
let endpoint = start_mock(state.clone()).await;
|
||||
let calls = Arc::new(AtomicU32::new(0));
|
||||
let mut config = ClientConfig::new(&endpoint);
|
||||
config.auth = Some(AuthConfig::Oidc {
|
||||
token: "stale-token".to_string(),
|
||||
expires_at: Some(
|
||||
std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.unwrap()
|
||||
.as_secs()
|
||||
+ 5,
|
||||
),
|
||||
refresh: Some(Arc::new(OneShotRefresher {
|
||||
calls: calls.clone(),
|
||||
})),
|
||||
});
|
||||
let client = OpenShellClient::connect(config).await.unwrap();
|
||||
let mut updates = client.raw_sandbox_updates();
|
||||
let request = proto::UpdateSandboxRequest {
|
||||
workspace_scope: Some(proto::workspace_selector("team")),
|
||||
name: "sandbox".into(),
|
||||
image: Some("example/workload:new".into()),
|
||||
expected_resource_version: 42,
|
||||
request_id: "550e8400-e29b-41d4-a716-446655440000".into(),
|
||||
..Default::default()
|
||||
};
|
||||
assert_eq!(
|
||||
updates
|
||||
.update_sandbox(request.clone())
|
||||
.await
|
||||
.unwrap_err()
|
||||
.code(),
|
||||
tonic::Code::Unauthenticated
|
||||
);
|
||||
assert_eq!(calls.load(Ordering::SeqCst), 0);
|
||||
client
|
||||
.raw_sandbox_updates_fresh()
|
||||
.await
|
||||
.unwrap()
|
||||
.update_sandbox(request.clone())
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(calls.load(Ordering::SeqCst), 1);
|
||||
// Previously created raw clients read the same refreshed bearer slot.
|
||||
updates.update_sandbox(request.clone()).await.unwrap();
|
||||
client
|
||||
.raw_grpc()
|
||||
.health(proto::HealthRequest {})
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(*state.last_update.lock().await, Some(request));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn watch_logs_forwards_logs_and_warnings() {
|
||||
let state = Arc::new(MockState {
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
//! Compile-time source-compatibility fixture for the v0.1.5 public status shape.
|
||||
|
||||
use openshell_sdk::raw::proto::SandboxStatus;
|
||||
|
||||
#[test]
|
||||
fn baseline_exhaustive_sandbox_status_literal_still_compiles() {
|
||||
// Intentionally no `..Default::default()`: adding even an optional public
|
||||
// field breaks existing downstream Rust exhaustive struct literals.
|
||||
let status = SandboxStatus {
|
||||
agent_pod: String::new(),
|
||||
agent_fd: String::new(),
|
||||
sandbox_fd: String::new(),
|
||||
conditions: Vec::new(),
|
||||
phase: 0,
|
||||
current_policy_version: 0,
|
||||
main_process_instance_id: String::new(),
|
||||
exit_code: None,
|
||||
endpoint_statuses: Vec::new(),
|
||||
configuration_admission: None,
|
||||
configuration_activated: None,
|
||||
provisioning: None,
|
||||
restart_count: 0,
|
||||
next_restart_time: None,
|
||||
main_process_started_time: None,
|
||||
last_token_grant_failure: None,
|
||||
};
|
||||
assert_eq!(status, SandboxStatus::default());
|
||||
}
|
||||
@@ -123,6 +123,50 @@ or two minutes after the withdrawal, after their owner was told to stop them.
|
||||
Pool members still authenticate with the one shared supervisor token, so a
|
||||
member identity is asserted by its hello rather than bound to its Pod.
|
||||
|
||||
## Optional in-process sandbox updates
|
||||
|
||||
`SharedComputeDriver` retains the generated baseline compute-driver trait, and
|
||||
`ComputeDriverInstance` retains its existing variants. Existing compiled drivers
|
||||
and factories need no new trait implementation or registration method.
|
||||
|
||||
A supporting factory overrides `ComputeDriverFactory::build_with_configuration_updates`
|
||||
to return its ordinary instance plus a separately registered updater. The default
|
||||
calls `build` and returns no updater. External drivers serve
|
||||
`ComputeDriverConfigurationUpdates` on their existing endpoint. Workload updates
|
||||
require `openshell.compute.update-sandbox-configuration` for image, command, or
|
||||
environment changes. Metadata-only replacements use one database CAS and update
|
||||
the label index without invoking a driver.
|
||||
|
||||
New updates require operator activation and compatible gateway membership.
|
||||
Accepted updates recover from private durable intent at startup and in periodic
|
||||
reconciliation, including when activation is subsequently disabled. Recovery
|
||||
pages through a partial database index of pending private update records instead
|
||||
of scanning sandboxes or completed receipts. Acceptance and completion maintain
|
||||
the indexed status in the same transaction as the intent and lifecycle interlock.
|
||||
Each gateway keeps automatic recovery backoff in memory. Failed attempts wait
|
||||
1, 2, 4, 8, then 15 minutes; each failure log includes the attempt and retry delay.
|
||||
Replicas retry independently, and a gateway restart resets its backoff. A new
|
||||
operation starts without delay. Completed or deleted updates are removed from
|
||||
the backoff map as recovery observes them.
|
||||
An authorized `UpdateSandbox` with no selected fields and the current resource
|
||||
version resumes immediately, bypassing backoff without reconstructing its selections. Completion retains the
|
||||
exact operation receipt for idempotent completion across replicas; a newer update
|
||||
invalidates it. Ambiguous results retain the baseline provisioning interlock.
|
||||
Explicit deletion can supersede a pending update: the driver only patches the
|
||||
original immutable resource and never creates a replacement. The Deleting CAS
|
||||
prevents delayed completion. Cleanup bypasses the pending-driver guard only when
|
||||
the pending operation ID matches the persisted update intent. Final removal
|
||||
deletes the sandbox and private intent atomically, retaining that evidence if
|
||||
cleanup fails or loses a resource-version race.
|
||||
|
||||
Metadata replacement preserves gateway-owned keys, including retention, while
|
||||
other user keys under OpenShell domains remain editable. Completion preserves
|
||||
later annotation merges accepted through `UpdateConfig`.
|
||||
|
||||
A provisioning timeout can be repaired by Stop after its original driver work
|
||||
settles and cleanup completes. Stop exposes Stopped without discarding the timeout
|
||||
diagnosis or storage. This lifecycle behavior also applies when updates are disabled.
|
||||
|
||||
## Operator credential export
|
||||
|
||||
`GetProviderCredentials` exports selected profile-declared runtime environment
|
||||
|
||||
@@ -30,8 +30,10 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
let storage_proto_dir = manifest_dir.join("proto");
|
||||
let public_proto_dir = manifest_dir.join("../../proto");
|
||||
let storage_proto = storage_proto_dir.join("storage.proto");
|
||||
let image_proto = storage_proto_dir.join("sandbox_image.proto");
|
||||
|
||||
println!("cargo:rerun-if-changed={}", storage_proto.display());
|
||||
println!("cargo:rerun-if-changed={}", image_proto.display());
|
||||
for imported_proto in [
|
||||
"datamodel.proto",
|
||||
"openshell.proto",
|
||||
@@ -71,7 +73,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
"::openshell_core::proto::sandbox::v1",
|
||||
)
|
||||
.file_descriptor_set_path(&descriptor_path)
|
||||
.compile_with_config(proto_config, &[storage_proto], &include_dirs)?;
|
||||
.compile_with_config(proto_config, &[storage_proto, image_proto], &include_dirs)?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
CREATE INDEX IF NOT EXISTS sandbox_pending_updates_idx
|
||||
ON objects (object_type, id)
|
||||
WHERE object_type = 'sandbox_image_state' AND status = 'pending';
|
||||
@@ -0,0 +1,3 @@
|
||||
CREATE INDEX IF NOT EXISTS sandbox_pending_updates_idx
|
||||
ON objects (object_type, id)
|
||||
WHERE object_type = 'sandbox_image_state' AND status = 'pending';
|
||||
@@ -0,0 +1,32 @@
|
||||
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
syntax = "proto3";
|
||||
|
||||
// A new private format; frozen openshell.storage.v1 remains unchanged.
|
||||
package openshell.storage.image.v1;
|
||||
import "openshell.proto";
|
||||
|
||||
// Gateway-private image mutation state. Never embed this in public SandboxStatus:
|
||||
// older gateways may rewrite public protobufs without retaining unknown fields.
|
||||
message StoredSandboxImageUpdate {
|
||||
string operation_id = 1;
|
||||
string previous_image = 2;
|
||||
string image = 3;
|
||||
uint64 revision = 4;
|
||||
// Presence retains selection semantics.
|
||||
openshell.v1.SandboxCommandUpdate command = 5;
|
||||
openshell.v1.SandboxMapUpdate environment = 6;
|
||||
openshell.v1.SandboxMapUpdate labels = 7;
|
||||
openshell.v1.SandboxMapUpdate annotations = 8;
|
||||
}
|
||||
|
||||
message StoredSandboxImageState {
|
||||
string sandbox_id = 1;
|
||||
uint64 revision = 2;
|
||||
StoredSandboxImageUpdate pending = 3;
|
||||
// Original presence and attempt identity while the interlock is held.
|
||||
// Completion preserves newer diagnostics and clock edits on the live record.
|
||||
openshell.v1.SandboxProvisioning previous_provisioning = 4;
|
||||
// Exact last successful intent, for idempotent completion across replicas.
|
||||
StoredSandboxImageUpdate completed = 5;
|
||||
}
|
||||
@@ -313,6 +313,11 @@ struct RunArgs {
|
||||
action = ArgAction::Set
|
||||
)]
|
||||
enable_websocket_tunnel: bool,
|
||||
|
||||
/// Activate stopped-sandbox configuration updates only after retiring every incompatible gateway
|
||||
/// and settling its outstanding driver operations. See HA upgrade guidance.
|
||||
#[arg(long, env = "OPENSHELL_ENABLE_SANDBOX_UPDATES", default_value_t = false, action = ArgAction::Set)]
|
||||
enable_sandbox_updates: bool,
|
||||
}
|
||||
|
||||
pub fn command() -> Command {
|
||||
@@ -594,6 +599,8 @@ fn prepare_server_config_with_drivers(
|
||||
.service_routing
|
||||
.validate(&config.name)
|
||||
.map_err(|error| miette::miette!("invalid service routing configuration: {error}"))?;
|
||||
|
||||
config.enable_sandbox_updates = args.enable_sandbox_updates;
|
||||
if let Some(sources) = file
|
||||
.as_ref()
|
||||
.and_then(|file| file.openshell.gateway.provider_profile_sources.clone())
|
||||
@@ -1386,6 +1393,11 @@ fn merge_file_into_args(args: &mut RunArgs, file: &GatewayFileSection, matches:
|
||||
{
|
||||
args.enable_websocket_tunnel = enabled;
|
||||
}
|
||||
if let Some(enabled) = file.enable_sandbox_updates
|
||||
&& arg_defaulted(matches, "enable_sandbox_updates")
|
||||
{
|
||||
args.enable_sandbox_updates = enabled;
|
||||
}
|
||||
if let Some(mtls_auth) = &file.mtls_auth
|
||||
&& arg_defaulted(matches, "enable_mtls_auth")
|
||||
{
|
||||
@@ -3102,6 +3114,40 @@ mod tests {
|
||||
toml::from_str(toml).expect("valid TOML in test fixture")
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sandbox_update_activation_defaults_off_and_file_requires_explicit_opt_in() {
|
||||
Environment::new().remove("OPENSHELL_ENABLE_SANDBOX_UPDATES").run(|| {
|
||||
let (mut args, matches) = parse_with_args(&["openshell-gateway"]);
|
||||
assert!(!args.enable_sandbox_updates);
|
||||
assert!(!openshell_core::Config::new(None).enable_sandbox_updates);
|
||||
let file = config_file_from_toml("[openshell]\nversion = 2\n[openshell.gateway]\nenable_sandbox_updates = true\n");
|
||||
merge_file_into_args(&mut args, &file.openshell.gateway, &matches);
|
||||
assert!(args.enable_sandbox_updates);
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn explicit_cli_or_environment_disable_overrides_file_activation() {
|
||||
let file = config_file_from_toml(
|
||||
"[openshell]\nversion = 2\n[openshell.gateway]\nenable_sandbox_updates = true\n",
|
||||
);
|
||||
Environment::new()
|
||||
.remove("OPENSHELL_ENABLE_SANDBOX_UPDATES")
|
||||
.run(|| {
|
||||
let (mut args, matches) =
|
||||
parse_with_args(&["openshell-gateway", "--enable-sandbox-updates", "false"]);
|
||||
merge_file_into_args(&mut args, &file.openshell.gateway, &matches);
|
||||
assert!(!args.enable_sandbox_updates);
|
||||
});
|
||||
Environment::new()
|
||||
.set("OPENSHELL_ENABLE_SANDBOX_UPDATES", "false")
|
||||
.run(|| {
|
||||
let (mut args, matches) = parse_with_args(&["openshell-gateway"]);
|
||||
merge_file_into_args(&mut args, &file.openshell.gateway, &matches);
|
||||
assert!(!args.enable_sandbox_updates);
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_legacy_drivers_flag() {
|
||||
let error = command()
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -4,6 +4,7 @@
|
||||
//! Gateway-owned compute orchestration over a pluggable compute backend.
|
||||
|
||||
pub mod driver_config;
|
||||
mod image_update;
|
||||
pub mod lease;
|
||||
mod mutation_guard;
|
||||
pub mod provisioning_deadline;
|
||||
@@ -86,6 +87,13 @@ pub type DriverWatchStream =
|
||||
pub type SharedComputeDriver =
|
||||
Arc<dyn ComputeDriver<WatchSandboxesStream = DriverWatchStream> + Send + Sync>;
|
||||
|
||||
/// Optional configuration-update registration, independent of the baseline driver object.
|
||||
pub type SharedComputeDriverConfigurationUpdates = Arc<
|
||||
dyn openshell_core::compute::ComputeDriver<WatchSandboxesStream = DriverWatchStream>
|
||||
+ Send
|
||||
+ Sync,
|
||||
>;
|
||||
|
||||
fn supports_isolation_registration(capabilities: &[String]) -> bool {
|
||||
capabilities.iter().any(|capability| {
|
||||
capability == openshell_core::extension_protocol::COMPUTE_ISOLATION_BACKEND
|
||||
@@ -429,21 +437,26 @@ impl QuiesceOperation {
|
||||
let phase = SandboxPhase::try_from(sandbox.phase()).unwrap_or(SandboxPhase::Unknown);
|
||||
match self {
|
||||
Self::Stop => {
|
||||
if matches!(phase, SandboxPhase::Stopped | SandboxPhase::Completed)
|
||||
|| is_failed_main_process_result(sandbox)
|
||||
{
|
||||
if phase == SandboxPhase::Stopped {
|
||||
return Ok(QuiesceAdmission::Settled);
|
||||
}
|
||||
// Timeout cleanup owns reclamation while the record remains
|
||||
// Error. Its later start may recreate missing compute; an
|
||||
// ordinary stop must not replace that recovery state with
|
||||
// Stopping or Stopped.
|
||||
if phase == SandboxPhase::Error && provisioning_deadline::timed_out(sandbox) {
|
||||
// Timeout cleanup owns reclamation until it completes. Once
|
||||
// reclaimed, explicit stop can expose Stopped for image repair
|
||||
// while preserving the timeout record for a later start.
|
||||
if phase == SandboxPhase::Error
|
||||
&& provisioning_deadline::timed_out(sandbox)
|
||||
&& sandbox
|
||||
.status
|
||||
.as_ref()
|
||||
.and_then(|status| status.provisioning.as_ref())
|
||||
.is_none_or(|record| record.cleanup_completed_time.is_none())
|
||||
{
|
||||
return Err(Status::failed_precondition(
|
||||
"sandbox provisioning timed out; retry start after timeout cleanup completes",
|
||||
"sandbox provisioning timed out; wait for timeout cleanup before stop or start",
|
||||
));
|
||||
}
|
||||
// Error does not prove compute has stopped. Use the normal
|
||||
// Neither a main-process exit nor an infrastructure Error
|
||||
// proves compute has stopped. Use the normal
|
||||
// driver stop and session cleanup before a subsequent start
|
||||
// renews authentication and repeats configuration admission
|
||||
// for the retained sandbox. Stopping a suspended sandbox
|
||||
@@ -452,6 +465,7 @@ impl QuiesceOperation {
|
||||
if !matches!(
|
||||
phase,
|
||||
SandboxPhase::Ready
|
||||
| SandboxPhase::Completed
|
||||
| SandboxPhase::Stopping
|
||||
| SandboxPhase::Error
|
||||
| SandboxPhase::Suspending
|
||||
@@ -459,7 +473,7 @@ impl QuiesceOperation {
|
||||
) && !is_automatic_restart_transition(sandbox)
|
||||
{
|
||||
return Err(Status::failed_precondition(format!(
|
||||
"sandbox must be Ready, Suspended, Error, or in an automatic restart to stop (current phase: {phase:?})"
|
||||
"sandbox must be Ready, Completed, Suspended, Error, or in an automatic restart to stop (current phase: {phase:?})"
|
||||
)));
|
||||
}
|
||||
Ok(if phase == SandboxPhase::Stopping {
|
||||
@@ -670,6 +684,7 @@ impl AcquiredRemoteDriverEndpoint {
|
||||
#[derive(Debug, Clone)]
|
||||
struct RemoteComputeDriver {
|
||||
client: RemoteComputeDriverClient,
|
||||
configuration_updates_client: openshell_core::proto::compute::v1::compute_driver_configuration_updates_client::ComputeDriverConfigurationUpdatesClient<tonic::service::interceptor::InterceptedService<Channel, TraceContextInterceptor>>,
|
||||
admission_acknowledgement: Arc<StdMutex<Option<String>>>,
|
||||
}
|
||||
|
||||
@@ -680,7 +695,8 @@ type RemoteComputeDriverClient = ComputeDriverClient<
|
||||
impl RemoteComputeDriver {
|
||||
fn new(channel: Channel) -> Self {
|
||||
Self {
|
||||
client: ComputeDriverClient::with_interceptor(channel, TraceContextInterceptor),
|
||||
client: ComputeDriverClient::with_interceptor(channel.clone(), TraceContextInterceptor),
|
||||
configuration_updates_client: openshell_core::proto::compute::v1::compute_driver_configuration_updates_client::ComputeDriverConfigurationUpdatesClient::with_interceptor(channel, TraceContextInterceptor),
|
||||
admission_acknowledgement: Arc::new(StdMutex::new(None)),
|
||||
}
|
||||
}
|
||||
@@ -721,6 +737,23 @@ impl RemoteComputeDriver {
|
||||
}
|
||||
}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl openshell_core::compute::ComputeDriver for RemoteComputeDriver {
|
||||
async fn update_sandbox_configuration(
|
||||
&self,
|
||||
request: Request<openshell_core::proto::compute::v1::UpdateSandboxConfigurationRequest>,
|
||||
) -> Result<
|
||||
tonic::Response<openshell_core::proto::compute::v1::UpdateSandboxConfigurationResponse>,
|
||||
Status,
|
||||
> {
|
||||
self.verify_admission_policy(false).await?;
|
||||
self.configuration_updates_client
|
||||
.clone()
|
||||
.update_sandbox_configuration(request)
|
||||
.await
|
||||
}
|
||||
}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl ComputeDriver for RemoteComputeDriver {
|
||||
type WatchSandboxesStream = DriverWatchStream;
|
||||
@@ -877,11 +910,15 @@ pub struct ComputeRuntime {
|
||||
admission: openshell_core::resource_admission::DriverAdmissionConfig,
|
||||
admission_acknowledgement: String,
|
||||
driver: TracedDriver,
|
||||
image_update_driver: Option<SharedComputeDriverConfigurationUpdates>,
|
||||
driver_info: ComputeDriverInfoSnapshot,
|
||||
telemetry_compute_driver: TelemetryComputeDriver,
|
||||
driver_process: Option<Arc<ManagedDriverProcess>>,
|
||||
default_image: String,
|
||||
image_preparation_timeout_seconds: u32,
|
||||
sandbox_updates_enabled: bool,
|
||||
update_recovery_sweep: Arc<tokio::sync::Semaphore>,
|
||||
update_recovery_backoff: Arc<Mutex<HashMap<String, image_update::UpdateRecoveryBackoff>>>,
|
||||
store: Arc<Store>,
|
||||
sandbox_index: SandboxIndex,
|
||||
sandbox_watch_bus: SandboxWatchBus,
|
||||
@@ -991,6 +1028,7 @@ impl ComputeRuntime {
|
||||
Ok(Self {
|
||||
isolation_backend: None,
|
||||
driver: TracedDriver::new(driver, driver_name),
|
||||
image_update_driver: None,
|
||||
admission: openshell_core::resource_admission::DriverAdmissionConfig::default(),
|
||||
admission_acknowledgement: capabilities.resource_admission_policy,
|
||||
driver_info,
|
||||
@@ -998,6 +1036,9 @@ impl ComputeRuntime {
|
||||
driver_process,
|
||||
default_image,
|
||||
image_preparation_timeout_seconds: 1800,
|
||||
sandbox_updates_enabled: false,
|
||||
update_recovery_sweep: Arc::new(tokio::sync::Semaphore::new(1)),
|
||||
update_recovery_backoff: Arc::default(),
|
||||
store,
|
||||
sandbox_index,
|
||||
sandbox_watch_bus,
|
||||
@@ -1129,10 +1170,10 @@ impl ComputeRuntime {
|
||||
tracing_log_bus: TracingLogBus,
|
||||
supervisor_sessions: Arc<SupervisorSessionRegistry>,
|
||||
) -> Result<Self, ComputeError> {
|
||||
let driver: SharedComputeDriver = Arc::new(RemoteComputeDriver::new(endpoint.channel));
|
||||
let driver = Arc::new(RemoteComputeDriver::new(endpoint.channel));
|
||||
Self::from_driver(
|
||||
endpoint.name,
|
||||
driver,
|
||||
driver.clone(),
|
||||
endpoint.driver_process,
|
||||
store,
|
||||
sandbox_index,
|
||||
@@ -1141,6 +1182,15 @@ impl ComputeRuntime {
|
||||
supervisor_sessions,
|
||||
)
|
||||
.await
|
||||
.map(|runtime| runtime.with_image_update_driver(Some(driver)))
|
||||
}
|
||||
|
||||
pub(crate) fn with_image_update_driver(
|
||||
mut self,
|
||||
driver: Option<SharedComputeDriverConfigurationUpdates>,
|
||||
) -> Self {
|
||||
self.image_update_driver = driver;
|
||||
self
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
@@ -1808,14 +1858,15 @@ impl ComputeRuntime {
|
||||
Some(guard) => guard,
|
||||
None => runtime.lock_sandbox_for_lifecycle(&lifecycle_guard).await,
|
||||
};
|
||||
runtime
|
||||
.compensate_failed_create_inner(
|
||||
&created,
|
||||
lifecycle_guard,
|
||||
sandbox_guard,
|
||||
original,
|
||||
)
|
||||
.await
|
||||
// Keep compensation's future bounded as the persisted sandbox
|
||||
// status grows (including image-update reconciliation state).
|
||||
Box::pin(runtime.compensate_failed_create_inner(
|
||||
&created,
|
||||
lifecycle_guard,
|
||||
sandbox_guard,
|
||||
original,
|
||||
))
|
||||
.await
|
||||
}
|
||||
.instrument(request_span),
|
||||
)
|
||||
@@ -2042,14 +2093,32 @@ impl ComputeRuntime {
|
||||
QuiesceAdmission::Begin => {}
|
||||
}
|
||||
|
||||
let (reason, message) = operation.requested_condition();
|
||||
let reclaimed_timeout = operation == QuiesceOperation::Stop
|
||||
&& current.phase() == SandboxPhase::Error as i32
|
||||
&& provisioning_deadline::timed_out(¤t);
|
||||
let (next_phase, reason, message) = if reclaimed_timeout {
|
||||
(
|
||||
SandboxPhase::Stopped,
|
||||
"StoppedAfterProvisioningTimeout",
|
||||
"Timed-out compute has been reclaimed; sandbox stopped for repair",
|
||||
)
|
||||
} else {
|
||||
let (reason, message) = operation.requested_condition();
|
||||
(operation.in_flight_phase(), reason, message)
|
||||
};
|
||||
match self
|
||||
.write_lifecycle_phase(¤t, operation.in_flight_phase(), reason, message)
|
||||
.write_lifecycle_phase(¤t, next_phase, reason, message)
|
||||
.await
|
||||
{
|
||||
Ok(transition) => {
|
||||
self.sandbox_index.update_from_sandbox(&transition);
|
||||
self.sandbox_watch_bus.notify(&sandbox_id);
|
||||
if reclaimed_timeout {
|
||||
self.cleanup_stopped_sandbox_sessions(&transition)
|
||||
.await
|
||||
.map_err(Status::internal)?;
|
||||
return Ok(transition);
|
||||
}
|
||||
break (current, transition);
|
||||
}
|
||||
Err(crate::persistence::PersistenceError::Conflict { .. })
|
||||
@@ -2134,8 +2203,12 @@ impl ComputeRuntime {
|
||||
Ok(settled)
|
||||
}
|
||||
Err(err) => {
|
||||
self.recover_failed_lifecycle(&lifecycle_guard, &transition, &previous, true)
|
||||
.await;
|
||||
if let Some(terminal) = self
|
||||
.recover_failed_lifecycle(&lifecycle_guard, &transition, &previous, true)
|
||||
.await
|
||||
{
|
||||
return Ok(terminal);
|
||||
}
|
||||
Err(Status::new(
|
||||
err.code(),
|
||||
format!("{} sandbox failed: {}", operation.verb(), err.message()),
|
||||
@@ -2264,6 +2337,7 @@ impl ComputeRuntime {
|
||||
self.validate_launch_signer_configured(authority.is_some())?;
|
||||
}
|
||||
provisioning_operation::ensure_operation_settled(&candidate)?;
|
||||
self.ensure_no_pending_image_update(&candidate).await?;
|
||||
if provisioning_deadline::timed_out(&candidate)
|
||||
&& candidate
|
||||
.status
|
||||
@@ -2301,6 +2375,7 @@ impl ComputeRuntime {
|
||||
let mut attempts = 0;
|
||||
let (previous, starting, launch_authentication) = loop {
|
||||
provisioning_operation::ensure_operation_settled(¤t)?;
|
||||
self.ensure_no_pending_image_update(¤t).await?;
|
||||
let phase = SandboxPhase::try_from(current.phase()).unwrap_or(SandboxPhase::Unknown);
|
||||
if let StartIntent::Resume { .. } = intent {
|
||||
// Resume restores only retained state. A Starting record may
|
||||
@@ -2610,7 +2685,8 @@ impl ComputeRuntime {
|
||||
settled,
|
||||
}) => {
|
||||
provisioning_operation::ensure_current_result(&settled, &settled)?;
|
||||
self.recover_failed_lifecycle(&lifecycle_guard, &settled, &previous, false)
|
||||
let _ = self
|
||||
.recover_failed_lifecycle(&lifecycle_guard, &settled, &previous, false)
|
||||
.await;
|
||||
Err(Status::new(
|
||||
err.code(),
|
||||
@@ -2863,7 +2939,7 @@ impl ComputeRuntime {
|
||||
transition: &Sandbox,
|
||||
previous: &Sandbox,
|
||||
expected_stopped: bool,
|
||||
) {
|
||||
) -> Option<Sandbox> {
|
||||
let sandbox_id = transition.object_id();
|
||||
let sandbox_name = transition.object_name();
|
||||
let observed = tokio::time::timeout(
|
||||
@@ -2875,6 +2951,14 @@ impl ComputeRuntime {
|
||||
let _sandbox_guard = self.lock_sandbox_for_lifecycle(lifecycle_guard).await;
|
||||
|
||||
match observed {
|
||||
Ok(None) if terminal_stop_exit_code(transition).is_some() => {
|
||||
match self.restore_missing_terminal_stop(transition).await {
|
||||
Ok(terminal) => return Some(terminal),
|
||||
Err(error) => {
|
||||
warn!(sandbox_id, %error, "Failed to restore terminal result after stop");
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(Some(snapshot)) if snapshot.id == sandbox_id && snapshot.status.is_some() => {
|
||||
let backend_phase = derive_phase(snapshot.status.as_ref());
|
||||
let observed_stopped = backend_phase == SandboxPhase::Stopped
|
||||
@@ -2943,6 +3027,38 @@ impl ComputeRuntime {
|
||||
);
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Called under the sandbox mutation lock after an authoritative lookup
|
||||
/// confirms absent compute. The persisted exit code survives a gateway
|
||||
/// restart; Stopping only retains it when stop began from a terminal result.
|
||||
async fn restore_missing_terminal_stop(&self, transition: &Sandbox) -> Result<Sandbox, Status> {
|
||||
let exit_code = terminal_stop_exit_code(transition)
|
||||
.ok_or_else(|| Status::failed_precondition("stop has no terminal result to restore"))?;
|
||||
let instance_id = transition
|
||||
.status
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.main_process_instance_id
|
||||
.clone();
|
||||
let restored = self
|
||||
.store
|
||||
.update_message_cas::<Sandbox, _>(
|
||||
transition.object_id(),
|
||||
sandbox_resource_version(transition),
|
||||
|sandbox| apply_main_process_exit(sandbox, &instance_id, exit_code),
|
||||
)
|
||||
.await
|
||||
.map_err(|error| {
|
||||
crate::grpc::persistence_error_to_status(error, "restore terminal sandbox result")
|
||||
})?;
|
||||
self.sandbox_index.update_from_sandbox(&restored);
|
||||
self.sandbox_watch_bus.notify(restored.object_id());
|
||||
self.cleanup_stopped_sandbox_sessions(&restored)
|
||||
.await
|
||||
.map_err(Status::internal)?;
|
||||
Ok(restored)
|
||||
}
|
||||
|
||||
async fn reconcile_lifecycle_snapshot(
|
||||
@@ -3406,7 +3522,23 @@ impl ComputeRuntime {
|
||||
.map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))?
|
||||
.ok_or_else(|| Status::not_found("sandbox not found"))?,
|
||||
};
|
||||
provisioning_operation::ensure_operation_settled(&sandbox)?;
|
||||
// Explicit deletion can supersede a configuration update: the
|
||||
// driver only patches an existing immutable resource and cannot
|
||||
// recreate it after deletion. Keep the interlock until removal so
|
||||
// a failed delete can safely restore the pending update.
|
||||
let pending_update = self
|
||||
.store
|
||||
.get_sandbox_image_state(sandbox_id)
|
||||
.await
|
||||
.map_err(|_| Status::unavailable("cannot read pending sandbox update"))?
|
||||
.and_then(|state| state.pending);
|
||||
if failed_create.is_none()
|
||||
&& let Some(intent) = pending_update
|
||||
{
|
||||
image_update::ensure_image_update_interlock(&sandbox, &intent)?;
|
||||
} else {
|
||||
provisioning_operation::ensure_operation_settled(&sandbox)?;
|
||||
}
|
||||
|
||||
// Failed-create compensation owns only its original attempt.
|
||||
// Preserve its timeout diagnosis and any replacement attempt on
|
||||
@@ -3562,7 +3694,10 @@ impl ComputeRuntime {
|
||||
|
||||
let mut delete_resource_version = record.resource_version;
|
||||
let mut sandbox = decode_sandbox_record(&record)?;
|
||||
if provisioning_deadline::driver_operation_pending(&sandbox) {
|
||||
if self
|
||||
.pending_driver_operation_blocks_cleanup(&sandbox)
|
||||
.await?
|
||||
{
|
||||
return Ok(false);
|
||||
}
|
||||
if sandbox.phase() != SandboxPhase::Deleting as i32 {
|
||||
@@ -3588,10 +3723,11 @@ impl ComputeRuntime {
|
||||
return Err(error);
|
||||
}
|
||||
|
||||
match self
|
||||
.store
|
||||
.delete_if(Sandbox::object_type(), sandbox_id, delete_resource_version)
|
||||
.await
|
||||
match Box::pin(
|
||||
self.store
|
||||
.delete_sandbox_with_image_state_if(sandbox_id, delete_resource_version),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(true) => {
|
||||
self.cleanup_removed_sandbox_state(sandbox_id);
|
||||
@@ -4077,7 +4213,7 @@ impl ComputeRuntime {
|
||||
{
|
||||
Box::pin(self.recover_persisted_lifecycle_transitions()).await?;
|
||||
if !self.driver_info.gateway_manages_lifecycle {
|
||||
return Ok(());
|
||||
return self.recover_pending_sandbox_updates().await;
|
||||
}
|
||||
|
||||
let sandbox_ids = self.list_persisted_sandbox_ids("gateway startup").await?;
|
||||
@@ -4337,7 +4473,7 @@ impl ComputeRuntime {
|
||||
"Sandbox start sweep complete"
|
||||
);
|
||||
}
|
||||
Ok(())
|
||||
self.recover_pending_sandbox_updates().await
|
||||
}
|
||||
|
||||
async fn recover_persisted_lifecycle_transitions(&self) -> Result<(), String> {
|
||||
@@ -4358,7 +4494,10 @@ impl ComputeRuntime {
|
||||
continue;
|
||||
}
|
||||
};
|
||||
if provisioning_deadline::driver_operation_pending(&sandbox) {
|
||||
if self
|
||||
.pending_driver_operation_blocks_cleanup(&sandbox)
|
||||
.await?
|
||||
{
|
||||
warn!(
|
||||
sandbox_id,
|
||||
"Retaining pending driver operation during gateway recovery"
|
||||
@@ -4378,12 +4517,20 @@ impl ComputeRuntime {
|
||||
}
|
||||
}
|
||||
SandboxPhase::Stopping => {
|
||||
self.recover_quiesce_transition(QuiesceOperation::Stop, &sandbox)
|
||||
.await;
|
||||
self.recover_quiesce_transition(
|
||||
QuiesceOperation::Stop,
|
||||
&sandbox,
|
||||
&lifecycle_guard,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
SandboxPhase::Suspending => {
|
||||
self.recover_quiesce_transition(QuiesceOperation::Suspend, &sandbox)
|
||||
.await;
|
||||
self.recover_quiesce_transition(
|
||||
QuiesceOperation::Suspend,
|
||||
&sandbox,
|
||||
&lifecycle_guard,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
SandboxPhase::Starting => {
|
||||
if let Err(error) = self.validate_isolation_backend_binding(&sandbox) {
|
||||
@@ -4458,7 +4605,7 @@ impl ComputeRuntime {
|
||||
if let ProvisioningOperationError::Driver { settled, .. } = err
|
||||
&& in_flight_resume_intent(&settled).is_some()
|
||||
{
|
||||
Box::pin(self.recover_failed_lifecycle(
|
||||
let _ = Box::pin(self.recover_failed_lifecycle(
|
||||
&lifecycle_guard,
|
||||
&settled,
|
||||
&sandbox,
|
||||
@@ -4477,12 +4624,24 @@ impl ComputeRuntime {
|
||||
|
||||
/// Re-issue the idempotent driver call of a stop or suspend interrupted
|
||||
/// by a gateway restart, then commit its settled phase.
|
||||
async fn recover_quiesce_transition(&self, operation: QuiesceOperation, sandbox: &Sandbox) {
|
||||
async fn recover_quiesce_transition(
|
||||
&self,
|
||||
operation: QuiesceOperation,
|
||||
sandbox: &Sandbox,
|
||||
lifecycle_guard: &SandboxLifecycleGuard,
|
||||
) {
|
||||
let sandbox_id = sandbox.object_id();
|
||||
if let Err(err) = self
|
||||
.call_quiesce_driver(operation, sandbox_id, sandbox.object_name())
|
||||
.await
|
||||
{
|
||||
if terminal_stop_exit_code(sandbox).is_some()
|
||||
&& Box::pin(self.recover_failed_lifecycle(lifecycle_guard, sandbox, sandbox, true))
|
||||
.await
|
||||
.is_some()
|
||||
{
|
||||
return;
|
||||
}
|
||||
warn!(sandbox_id, operation = operation.verb(), error = %err, "Failed to recover sandbox lifecycle transition");
|
||||
return;
|
||||
}
|
||||
@@ -4767,6 +4926,9 @@ impl ComputeRuntime {
|
||||
|
||||
async fn reconcile_loop(self: Arc<Self>, mut cancel: watch::Receiver<bool>) {
|
||||
loop {
|
||||
if let Err(error) = self.recover_pending_sandbox_updates().await {
|
||||
warn!(%error, "Sandbox update recovery sweep failed");
|
||||
}
|
||||
if let Err(err) = self.reconcile_store_with_backend(ORPHAN_GRACE_PERIOD).await {
|
||||
warn!(error = %err, "Store reconciliation sweep failed");
|
||||
}
|
||||
@@ -5314,7 +5476,7 @@ impl ComputeRuntime {
|
||||
continue;
|
||||
}
|
||||
|
||||
self.prune_missing_sandbox(record, sweep_started_at_ms, grace_ms)
|
||||
Box::pin(self.prune_missing_sandbox(record, sweep_started_at_ms, grace_ms))
|
||||
.await
|
||||
.inspect_err(|_| crate::otel_tracing::mark_error(&tracing::Span::current()))?;
|
||||
}
|
||||
@@ -6083,9 +6245,17 @@ impl ComputeRuntime {
|
||||
.await
|
||||
.map_err(|e| e.to_string())?;
|
||||
if let Some(sandbox) = sandbox.as_ref() {
|
||||
if provisioning_deadline::driver_operation_pending(sandbox)
|
||||
// Timeout cleanup can emit Deleted before an explicit stop exposes
|
||||
// Stopped for repair. A delayed (or repeated) event must retain that
|
||||
// recovery record too; explicit deletion still transitions to Deleting.
|
||||
if self
|
||||
.pending_driver_operation_blocks_cleanup(sandbox)
|
||||
.await?
|
||||
|| (provisioning_deadline::timed_out(sandbox)
|
||||
&& sandbox.phase() == i32::from(SandboxPhase::Error))
|
||||
&& matches!(
|
||||
SandboxPhase::try_from(sandbox.phase()),
|
||||
Ok(SandboxPhase::Error | SandboxPhase::Stopped)
|
||||
))
|
||||
{
|
||||
return Ok(());
|
||||
}
|
||||
@@ -6121,7 +6291,10 @@ impl ComputeRuntime {
|
||||
sandbox: &Sandbox,
|
||||
expected_resource_version: u64,
|
||||
) -> Result<(), String> {
|
||||
if provisioning_deadline::driver_operation_pending(sandbox) {
|
||||
if self
|
||||
.pending_driver_operation_blocks_cleanup(sandbox)
|
||||
.await?
|
||||
{
|
||||
return Ok(());
|
||||
}
|
||||
let sandbox_id = sandbox.object_id();
|
||||
@@ -6131,6 +6304,8 @@ impl ComputeRuntime {
|
||||
}
|
||||
|
||||
async fn cleanup_sandbox_owned_records(&self, sandbox: &Sandbox) -> Result<(), String> {
|
||||
// Keep private update intent until atomic sandbox removal: a cleanup
|
||||
// failure or CAS conflict must leave evidence for the deletion retry.
|
||||
self.cleanup_ssh_identity(sandbox)
|
||||
.await
|
||||
.map_err(|error| error.to_string())?;
|
||||
@@ -6446,7 +6621,10 @@ impl ComputeRuntime {
|
||||
}
|
||||
|
||||
let sandbox = decode_sandbox_record(¤t_record)?;
|
||||
if provisioning_deadline::driver_operation_pending(&sandbox) {
|
||||
if self
|
||||
.pending_driver_operation_blocks_cleanup(&sandbox)
|
||||
.await?
|
||||
{
|
||||
return Ok(());
|
||||
}
|
||||
let age_ms = openshell_core::time::now_ms()
|
||||
@@ -6489,6 +6667,13 @@ impl ComputeRuntime {
|
||||
|
||||
let sandbox = decode_sandbox_record(¤t_record)?;
|
||||
let phase = SandboxPhase::try_from(sandbox.phase()).unwrap_or(SandboxPhase::Unknown);
|
||||
if terminal_stop_exit_code(&sandbox).is_some() {
|
||||
return self
|
||||
.restore_missing_terminal_stop(&sandbox)
|
||||
.await
|
||||
.map(|_| ())
|
||||
.map_err(|error| error.to_string());
|
||||
}
|
||||
if phase == SandboxPhase::Completed
|
||||
|| is_failed_main_process_result(&sandbox)
|
||||
|| provisioning_deadline::timed_out(&sandbox)
|
||||
@@ -6684,6 +6869,16 @@ fn has_specific_infrastructure_error(sandbox: &Sandbox) -> bool {
|
||||
})
|
||||
}
|
||||
|
||||
fn terminal_stop_exit_code(sandbox: &Sandbox) -> Option<i32> {
|
||||
if sandbox.phase() != SandboxPhase::Stopping as i32
|
||||
|| provisioning_deadline::timed_out(sandbox)
|
||||
|| provisioning_deadline::driver_operation_pending(sandbox)
|
||||
{
|
||||
return None;
|
||||
}
|
||||
sandbox.status.as_ref()?.exit_code
|
||||
}
|
||||
|
||||
fn is_failed_main_process_result(sandbox: &Sandbox) -> bool {
|
||||
sandbox.phase() == SandboxPhase::Error as i32
|
||||
&& sandbox.status.as_ref().is_some_and(|status| {
|
||||
@@ -7951,6 +8146,9 @@ fn apply_lifecycle_phase(
|
||||
message: &str,
|
||||
image_preparation_timeout_seconds: u32,
|
||||
) {
|
||||
let preserve_exit_code = phase == SandboxPhase::Stopping
|
||||
&& (sandbox.phase() == SandboxPhase::Completed as i32
|
||||
|| is_failed_main_process_result(sandbox));
|
||||
sandbox.set_phase(phase as i32);
|
||||
// Every explicit lifecycle write ends the previous transition's resume
|
||||
// intent. A resuming start records a new one after this write.
|
||||
@@ -7966,7 +8164,11 @@ fn apply_lifecycle_phase(
|
||||
let status = sandbox.status.get_or_insert_with(Default::default);
|
||||
// Retain the previous instance id as a tombstone until the restarted
|
||||
// supervisor registers its new id.
|
||||
status.exit_code = None;
|
||||
// Explicit stop preserves an already-recorded terminal result for
|
||||
// diagnosis while the sandbox is repaired. A fresh start clears it.
|
||||
if !preserve_exit_code {
|
||||
status.exit_code = None;
|
||||
}
|
||||
status.restart_count = 0;
|
||||
set_next_restart_at_ms(status, 0);
|
||||
set_main_process_started_at_ms(status, 0);
|
||||
@@ -8382,6 +8584,7 @@ pub fn new_test_runtime_with_driver(
|
||||
ComputeRuntime {
|
||||
isolation_backend: None,
|
||||
driver: TracedDriver::new(driver, "test".to_string()),
|
||||
image_update_driver: None,
|
||||
admission: openshell_core::resource_admission::DriverAdmissionConfig {
|
||||
allow_driver_config: true,
|
||||
..Default::default()
|
||||
@@ -8403,6 +8606,9 @@ pub fn new_test_runtime_with_driver(
|
||||
driver_process: None,
|
||||
default_image: "openshell/sandbox:test".to_string(),
|
||||
image_preparation_timeout_seconds: 1800,
|
||||
sandbox_updates_enabled: false,
|
||||
update_recovery_sweep: Arc::new(tokio::sync::Semaphore::new(1)),
|
||||
update_recovery_backoff: Arc::default(),
|
||||
store,
|
||||
sandbox_index: SandboxIndex::new(),
|
||||
sandbox_watch_bus: SandboxWatchBus::new(),
|
||||
@@ -9262,6 +9468,9 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl openshell_core::compute::ComputeDriver for ControlledDriver {}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl ComputeDriver for ControlledDriver {
|
||||
async fn authenticate_sandbox(
|
||||
@@ -9578,6 +9787,7 @@ mod tests {
|
||||
ComputeRuntime {
|
||||
isolation_backend: None,
|
||||
driver: TracedDriver::new(driver, "test-driver".to_string()),
|
||||
image_update_driver: None,
|
||||
admission: openshell_core::resource_admission::DriverAdmissionConfig {
|
||||
allow_driver_config: true,
|
||||
..Default::default()
|
||||
@@ -9599,6 +9809,9 @@ mod tests {
|
||||
driver_process: None,
|
||||
default_image: "openshell/sandbox:test".to_string(),
|
||||
image_preparation_timeout_seconds: 1800,
|
||||
sandbox_updates_enabled: false,
|
||||
update_recovery_sweep: Arc::new(Semaphore::new(1)),
|
||||
update_recovery_backoff: Arc::default(),
|
||||
store,
|
||||
sandbox_index: SandboxIndex::new(),
|
||||
sandbox_watch_bus: SandboxWatchBus::new(),
|
||||
@@ -12555,6 +12768,9 @@ mod tests {
|
||||
#[derive(Debug, Default)]
|
||||
struct FailingDriver(TestDriver);
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl openshell_core::compute::ComputeDriver for FailingDriver {}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl ComputeDriver for FailingDriver {
|
||||
async fn authenticate_sandbox(
|
||||
@@ -15120,7 +15336,7 @@ mod tests {
|
||||
|
||||
tokio::time::timeout(
|
||||
Duration::from_secs(1),
|
||||
runtime.reconcile_store_with_backend(Duration::ZERO),
|
||||
Box::pin(runtime.reconcile_store_with_backend(Duration::ZERO)),
|
||||
)
|
||||
.await
|
||||
.expect("prune sweep blocked on the stuck driver delete call")
|
||||
@@ -17538,10 +17754,11 @@ mod tests {
|
||||
async fn remote_compute_driver_interceptor_propagates_every_rpc() {
|
||||
use crate::otel_tracing::test_exporter;
|
||||
use crate::test_support::FakeComputeDriver;
|
||||
use openshell_core::compute::ComputeDriver as _;
|
||||
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let socket_path = dir.path().join("compute-driver.sock");
|
||||
let driver = FakeComputeDriver::new();
|
||||
let driver = FakeComputeDriver::new().with_sandbox_image_updates();
|
||||
let _server = driver.serve_uds(&socket_path).unwrap();
|
||||
let endpoint = connect_remote_compute_driver("external-test", &socket_path)
|
||||
.await
|
||||
@@ -17595,6 +17812,21 @@ mod tests {
|
||||
.watch_sandboxes(Request::new(WatchSandboxesRequest {}))
|
||||
.await
|
||||
.unwrap();
|
||||
remote
|
||||
.update_sandbox_configuration(Request::new(
|
||||
openshell_core::proto::compute::v1::UpdateSandboxConfigurationRequest {
|
||||
preconditions: Some(
|
||||
openshell_core::proto::compute::v1::SandboxUpdatePreconditions {
|
||||
sandbox_id: sandbox.id.clone(),
|
||||
image: "example/workload:v2".into(),
|
||||
..Default::default()
|
||||
},
|
||||
),
|
||||
..Default::default()
|
||||
},
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
remote
|
||||
.delete_sandbox(Request::new(DeleteSandboxRequest {
|
||||
sandbox_id: sandbox.id,
|
||||
@@ -17612,7 +17844,7 @@ mod tests {
|
||||
let traceparents = driver.traceparents();
|
||||
assert_eq!(
|
||||
traceparents.len(),
|
||||
10,
|
||||
12,
|
||||
"the client interceptor should cover every RPC"
|
||||
);
|
||||
assert!(
|
||||
@@ -18223,12 +18455,175 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock"
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
let status = openshell_core::compute::ComputeDriver::update_sandbox_configuration(
|
||||
&remote,
|
||||
Request::new(
|
||||
openshell_core::proto::compute::v1::UpdateSandboxConfigurationRequest::default(),
|
||||
),
|
||||
)
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert_eq!(status.code(), Code::FailedPrecondition);
|
||||
assert_eq!(
|
||||
driver.calls(),
|
||||
vec![FakeComputeDriverCall::GetCapabilities; 3]
|
||||
vec![FakeComputeDriverCall::GetCapabilities; 4]
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[cfg(unix)]
|
||||
async fn remote_compute_driver_negotiates_optional_image_service() {
|
||||
use crate::test_support::{FakeComputeDriver, FakeComputeDriverCall};
|
||||
use openshell_core::GetResourceVersion as _;
|
||||
use openshell_core::extension_protocol::COMPUTE_UPDATE_SANDBOX_CONFIGURATION;
|
||||
|
||||
for supported in [false, true] {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let socket_path = dir.path().join("image-updates.sock");
|
||||
let driver = FakeComputeDriver::new();
|
||||
let driver = if supported {
|
||||
driver.with_sandbox_image_updates()
|
||||
} else {
|
||||
driver
|
||||
};
|
||||
let _server = driver.serve_uds(&socket_path).unwrap();
|
||||
let endpoint = connect_remote_compute_driver("external-test", &socket_path)
|
||||
.await
|
||||
.unwrap();
|
||||
let store = Arc::new(crate::persistence::test_store().await);
|
||||
crate::ensure_default_workspace(&store).await.unwrap();
|
||||
let mut runtime = ComputeRuntime::new_remote_driver(
|
||||
endpoint,
|
||||
store.clone(),
|
||||
SandboxIndex::new(),
|
||||
SandboxWatchBus::new(),
|
||||
TracingLogBus::new(),
|
||||
Arc::new(SupervisorSessionRegistry::new()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
runtime.sandbox_updates_enabled = true;
|
||||
assert!(
|
||||
!runtime
|
||||
.driver_info
|
||||
.negotiated_extension
|
||||
.supported_capabilities
|
||||
.iter()
|
||||
.any(|capability| capability
|
||||
== openshell_core::extension_protocol::COMPUTE_ISOLATION_BACKEND),
|
||||
"image-only updates do not require external isolation registration"
|
||||
);
|
||||
assert_eq!(
|
||||
runtime
|
||||
.driver_info
|
||||
.negotiated_extension
|
||||
.supported_capabilities
|
||||
.iter()
|
||||
.any(|capability| capability == COMPUTE_UPDATE_SANDBOX_CONFIGURATION),
|
||||
supported,
|
||||
);
|
||||
let mut sandbox = sandbox_record("sb-image", "image-sandbox", SandboxPhase::Stopped);
|
||||
sandbox.metadata.as_mut().unwrap().annotations.insert(
|
||||
COMPUTE_RUNTIME_IDENTITY_ANNOTATION.into(),
|
||||
"stable-runtime".into(),
|
||||
);
|
||||
sandbox.spec = Some(SandboxSpec {
|
||||
template: Some(SandboxTemplate {
|
||||
image: "example/workload:v1".into(),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
});
|
||||
store.put_message(&sandbox).await.unwrap();
|
||||
let sandbox = store
|
||||
.get_message::<Sandbox>("sb-image")
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
driver.clear_calls();
|
||||
let result = runtime
|
||||
.update_sandbox_image(
|
||||
"sb-image",
|
||||
"example/workload:v2".into(),
|
||||
sandbox.get_resource_version(),
|
||||
)
|
||||
.await;
|
||||
let requests = driver
|
||||
.calls()
|
||||
.into_iter()
|
||||
.filter_map(|call| match call {
|
||||
FakeComputeDriverCall::UpdateSandboxImage { request } => Some(request),
|
||||
_ => None,
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
if supported {
|
||||
let updated = result.unwrap();
|
||||
assert_eq!(
|
||||
updated.spec.unwrap().template.unwrap().image,
|
||||
"example/workload:v2"
|
||||
);
|
||||
assert_eq!(
|
||||
requests.len(),
|
||||
2,
|
||||
"preflight and mutation use the optional service"
|
||||
);
|
||||
assert!(requests[0].validate_only);
|
||||
assert!(!requests[1].validate_only);
|
||||
for request in &requests {
|
||||
assert_eq!(request.sandbox_id, "sb-image");
|
||||
assert_eq!(request.expected_runtime_identity, "stable-runtime");
|
||||
assert_eq!(request.expected_image, "example/workload:v1");
|
||||
assert_eq!(request.image, "example/workload:v2");
|
||||
}
|
||||
assert_eq!(requests[0].operation_id, requests[1].operation_id);
|
||||
} else {
|
||||
assert_eq!(result.unwrap_err().code(), Code::Unimplemented);
|
||||
assert!(
|
||||
requests.is_empty(),
|
||||
"capability admission must precede extension calls"
|
||||
);
|
||||
let unchanged = store
|
||||
.get_message::<Sandbox>("sb-image")
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert_eq!(unchanged, sandbox);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn in_process_driver_defaults_image_updates_to_unimplemented() {
|
||||
// NoopTestDriver implements only the baseline generated trait.
|
||||
let mut runtime = test_runtime(Arc::new(NoopTestDriver::default()))
|
||||
.await
|
||||
.with_sandbox_updates(true);
|
||||
runtime
|
||||
.driver_info
|
||||
.negotiated_extension
|
||||
.supported_capabilities
|
||||
.push(openshell_core::extension_protocol::COMPUTE_UPDATE_SANDBOX_CONFIGURATION.into());
|
||||
let mut sandbox = sandbox_record(
|
||||
"unsupported-update",
|
||||
"unsupported-update",
|
||||
SandboxPhase::Stopped,
|
||||
);
|
||||
sandbox.spec = Some(SandboxSpec {
|
||||
template: Some(SandboxTemplate {
|
||||
image: "example/workload:v1".into(),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
});
|
||||
runtime.store.put_message(&sandbox).await.unwrap();
|
||||
let error = runtime
|
||||
.update_sandbox_image("unsupported-update", "example/workload:v2".into(), 1)
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert_eq!(error.code(), Code::Unimplemented);
|
||||
assert!(runtime.image_update_driver.is_none());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn negotiated_launch_requirement_rejects_create_before_driver_or_store() {
|
||||
let driver = Arc::new(TestDriver {
|
||||
|
||||
@@ -16,6 +16,226 @@ async fn stored(runtime: &ComputeRuntime, id: &str) -> Sandbox {
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn terminal_stop_preserves_result_when_compute_is_missing() {
|
||||
for exit_code in [0, 1, 137] {
|
||||
let driver = ControlledDriver::new();
|
||||
driver.set_stop_outcome(ControlledLifecycleOutcome::NotFound);
|
||||
let runtime = test_runtime(driver.clone()).await;
|
||||
let mut sandbox = sandbox_record("terminal-id", "terminal", SandboxPhase::Ready);
|
||||
apply_main_process_exit(&mut sandbox, "exited-instance", exit_code);
|
||||
runtime.store.put_message(&sandbox).await.unwrap();
|
||||
let session = ssh_session_record("old-session", sandbox.object_id());
|
||||
runtime.store.put_message(&session).await.unwrap();
|
||||
register_test_supervisor_session(&runtime, sandbox.object_id());
|
||||
|
||||
for _ in 0..2 {
|
||||
let result = runtime.stop_sandbox("default", "terminal").await.unwrap();
|
||||
assert_eq!(result.phase(), sandbox.phase());
|
||||
assert_eq!(result.status.as_ref().unwrap().exit_code, Some(exit_code));
|
||||
assert_eq!(
|
||||
result.status.as_ref().unwrap().main_process_instance_id,
|
||||
"exited-instance"
|
||||
);
|
||||
assert_eq!(
|
||||
ready_condition(&result).unwrap().reason,
|
||||
ready_condition(&sandbox).unwrap().reason
|
||||
);
|
||||
assert_eq!(result.spec, sandbox.spec);
|
||||
assert!(!runtime.supervisor_sessions.has_session(sandbox.object_id()));
|
||||
assert!(
|
||||
runtime
|
||||
.store
|
||||
.get_message::<SshSession>(session.object_id())
|
||||
.await
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
|
||||
let record = runtime
|
||||
.store
|
||||
.get(Sandbox::object_type(), sandbox.object_id())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
runtime
|
||||
.prune_missing_sandbox(record, openshell_core::time::now_ms(), 0)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(stored(&runtime, sandbox.object_id()).await, result);
|
||||
}
|
||||
assert_eq!(driver.start_calls(), 0);
|
||||
assert_eq!(driver.delete_calls(), 0);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn terminal_stop_recovers_from_persisted_result_after_restart_or_pruning() {
|
||||
for exit_code in [0, 1, 137] {
|
||||
for startup in [false, true] {
|
||||
let driver = ControlledDriver::new();
|
||||
driver.set_stop_outcome(ControlledLifecycleOutcome::NotFound);
|
||||
let runtime = test_runtime(driver).await;
|
||||
let mut sandbox = sandbox_record("terminal-id", "terminal", SandboxPhase::Ready);
|
||||
apply_main_process_exit(&mut sandbox, "exited-instance", exit_code);
|
||||
runtime.store.put_message(&sandbox).await.unwrap();
|
||||
let terminal = stored(&runtime, sandbox.object_id()).await;
|
||||
// Simulate a gateway dying after persisting Stopping, with no
|
||||
// in-memory pre-stop snapshot available to either recovery path.
|
||||
runtime
|
||||
.write_lifecycle_phase(
|
||||
&terminal,
|
||||
SandboxPhase::Stopping,
|
||||
"StopRequested",
|
||||
"Stop requested",
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let session = ssh_session_record("old-session", sandbox.object_id());
|
||||
runtime.store.put_message(&session).await.unwrap();
|
||||
register_test_supervisor_session(&runtime, sandbox.object_id());
|
||||
|
||||
if startup {
|
||||
runtime
|
||||
.recover_persisted_lifecycle_transitions()
|
||||
.await
|
||||
.unwrap();
|
||||
} else {
|
||||
let record = runtime
|
||||
.store
|
||||
.get(Sandbox::object_type(), sandbox.object_id())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
runtime
|
||||
.prune_missing_sandbox(record, openshell_core::time::now_ms(), 0)
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
let result = stored(&runtime, sandbox.object_id()).await;
|
||||
assert_eq!(result.phase(), terminal.phase());
|
||||
assert_eq!(result.status.as_ref().unwrap().exit_code, Some(exit_code));
|
||||
assert_eq!(
|
||||
ready_condition(&result).unwrap().reason,
|
||||
ready_condition(&terminal).unwrap().reason
|
||||
);
|
||||
assert!(!runtime.supervisor_sessions.has_session(sandbox.object_id()));
|
||||
assert!(
|
||||
runtime
|
||||
.store
|
||||
.get_message::<SshSession>(session.object_id())
|
||||
.await
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn terminal_stop_keeps_uncertain_compute_pending_until_explicit_retry() {
|
||||
let driver = ControlledDriver::new();
|
||||
driver.set_stop_outcome(ControlledLifecycleOutcome::NotFound);
|
||||
driver.set_get_outcome(ControlledGetOutcome::Error("lookup unavailable"));
|
||||
let runtime = test_runtime(driver.clone()).await;
|
||||
let mut sandbox = sandbox_record("terminal-id", "terminal", SandboxPhase::Ready);
|
||||
apply_main_process_exit(&mut sandbox, "exited-instance", 0);
|
||||
runtime.store.put_message(&sandbox).await.unwrap();
|
||||
|
||||
assert_eq!(
|
||||
runtime
|
||||
.stop_sandbox("default", "terminal")
|
||||
.await
|
||||
.unwrap_err()
|
||||
.code(),
|
||||
Code::NotFound
|
||||
);
|
||||
let pending = stored(&runtime, sandbox.object_id()).await;
|
||||
assert_eq!(pending.phase(), SandboxPhase::Stopping as i32);
|
||||
assert_eq!(pending.status.as_ref().unwrap().exit_code, Some(0));
|
||||
runtime
|
||||
.recover_persisted_lifecycle_transitions()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(stored(&runtime, sandbox.object_id()).await, pending);
|
||||
|
||||
driver.set_get_outcome(ControlledGetOutcome::Missing);
|
||||
let result = runtime.stop_sandbox("default", "terminal").await.unwrap();
|
||||
assert_eq!(result.phase(), SandboxPhase::Completed as i32);
|
||||
assert_eq!(result.status.as_ref().unwrap().exit_code, Some(0));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn terminal_stop_recovery_does_not_overwrite_concurrent_changes() {
|
||||
let driver = ControlledDriver::new();
|
||||
driver.set_stop_outcome(ControlledLifecycleOutcome::NotFound);
|
||||
driver.block_get();
|
||||
let runtime = test_runtime(driver.clone()).await;
|
||||
let mut sandbox = sandbox_record("terminal-id", "terminal", SandboxPhase::Ready);
|
||||
apply_main_process_exit(&mut sandbox, "exited-instance", 0);
|
||||
runtime.store.put_message(&sandbox).await.unwrap();
|
||||
let stop = {
|
||||
let runtime = runtime.clone();
|
||||
tokio::spawn(async move { runtime.stop_sandbox("default", "terminal").await })
|
||||
};
|
||||
driver.get_started.notified().await;
|
||||
let pending = stored(&runtime, sandbox.object_id()).await;
|
||||
let deleting = runtime
|
||||
.write_lifecycle_phase(
|
||||
&pending,
|
||||
SandboxPhase::Deleting,
|
||||
"DeleteRequested",
|
||||
"Delete requested",
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
driver.release_get();
|
||||
|
||||
assert!(stop.await.unwrap().is_err());
|
||||
assert_eq!(stored(&runtime, sandbox.object_id()).await, deleting);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn stop_without_terminal_result_still_reports_missing_compute() {
|
||||
let driver = ControlledDriver::new();
|
||||
driver.set_stop_outcome(ControlledLifecycleOutcome::NotFound);
|
||||
let runtime = test_runtime(driver).await;
|
||||
let sandbox = sandbox_record("running-id", "running", SandboxPhase::Ready);
|
||||
runtime.store.put_message(&sandbox).await.unwrap();
|
||||
assert_eq!(
|
||||
runtime
|
||||
.stop_sandbox("default", "running")
|
||||
.await
|
||||
.unwrap_err()
|
||||
.code(),
|
||||
Code::NotFound
|
||||
);
|
||||
let pending = stored(&runtime, sandbox.object_id()).await;
|
||||
assert_eq!(pending.phase(), SandboxPhase::Stopping as i32);
|
||||
assert_eq!(pending.status.as_ref().unwrap().exit_code, None);
|
||||
runtime
|
||||
.recover_persisted_lifecycle_transitions()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(stored(&runtime, sandbox.object_id()).await, pending);
|
||||
let record = runtime
|
||||
.store
|
||||
.get(Sandbox::object_type(), sandbox.object_id())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
runtime
|
||||
.prune_missing_sandbox(record, openshell_core::time::now_ms(), 0)
|
||||
.await
|
||||
.unwrap();
|
||||
let failed = stored(&runtime, sandbox.object_id()).await;
|
||||
assert_eq!(failed.phase(), SandboxPhase::Error as i32);
|
||||
assert_eq!(
|
||||
ready_condition(&failed).unwrap().reason,
|
||||
"ComputeResourceMissing"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn failed_gateway_start_can_be_stopped_and_retried() {
|
||||
let driver = ControlledDriver::new();
|
||||
@@ -152,6 +372,10 @@ async fn pending_driver_operation_blocks_error_recovery_without_mutation() {
|
||||
let runtime = test_runtime(driver.clone()).await;
|
||||
let mut sandbox = error_sandbox_record("pending-id", "pending", "StartFailed");
|
||||
let mut operation = provisioning_deadline::new_record(openshell_core::time::now_ms());
|
||||
// Even a completion timestamp cannot override an unsettled original driver
|
||||
// operation. This also fences malformed/stale timeout cleanup evidence.
|
||||
operation.timeout_time = openshell_core::time::timestamp_from_millis(300_000).ok();
|
||||
operation.cleanup_completed_time = openshell_core::time::timestamp_from_millis(320_000).ok();
|
||||
operation.driver_operation_pending = true;
|
||||
operation.driver_operation_id = "owned-operation".into();
|
||||
sandbox.status.as_mut().unwrap().provisioning = Some(operation);
|
||||
@@ -188,14 +412,39 @@ async fn timeout_stop_preserves_cleanup_and_direct_start_recovery() {
|
||||
runtime.store.put_message(&sandbox).await.unwrap();
|
||||
let before = stored(&runtime, sandbox.object_id()).await;
|
||||
|
||||
let error = runtime
|
||||
.stop_sandbox("default", "timeout")
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert_eq!(error.code(), Code::FailedPrecondition);
|
||||
assert!(error.message().contains("start"));
|
||||
assert_eq!(stored(&runtime, sandbox.object_id()).await, before);
|
||||
assert_eq!(driver.stop_calls(), 0, "timeout cleanup owns reclamation");
|
||||
let stop = runtime.stop_sandbox("default", "timeout").await;
|
||||
if cleanup_complete {
|
||||
let stopped = stop.expect(
|
||||
"completed timeout cleanup permits an explicit stopped state for image repair",
|
||||
);
|
||||
assert_eq!(stopped.phase(), SandboxPhase::Stopped as i32);
|
||||
assert_eq!(stopped.object_id(), before.object_id());
|
||||
assert_eq!(stopped.spec, before.spec);
|
||||
assert_eq!(
|
||||
stopped.status.as_ref().unwrap().provisioning,
|
||||
before.status.as_ref().unwrap().provisioning
|
||||
);
|
||||
let repeated = runtime.stop_sandbox("default", "timeout").await.unwrap();
|
||||
assert_eq!(
|
||||
repeated, stopped,
|
||||
"the retained stopped transition is idempotent"
|
||||
);
|
||||
// Docker cleanup and its snapshot poll can both report deletion of
|
||||
// the cancelled preparation after explicit stop has returned.
|
||||
for _ in 0..2 {
|
||||
runtime.apply_deleted(sandbox.object_id()).await.unwrap();
|
||||
assert_eq!(stored(&runtime, sandbox.object_id()).await, stopped);
|
||||
}
|
||||
} else {
|
||||
let error = stop.unwrap_err();
|
||||
assert_eq!(error.code(), Code::FailedPrecondition);
|
||||
assert_eq!(stored(&runtime, sandbox.object_id()).await, before);
|
||||
}
|
||||
assert_eq!(
|
||||
driver.stop_calls(),
|
||||
0,
|
||||
"timeout cleanup already owns reclamation"
|
||||
);
|
||||
|
||||
let result = runtime.start_sandbox("default", "timeout").await;
|
||||
if cleanup_complete {
|
||||
@@ -214,6 +463,35 @@ async fn timeout_stop_preserves_cleanup_and_direct_start_recovery() {
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn reclaimed_timeout_can_still_start_directly_without_stop() {
|
||||
let driver = ControlledDriver::new();
|
||||
driver.set_start_outcome(ControlledLifecycleOutcome::NotFound);
|
||||
driver.track_compute.store(true, Ordering::SeqCst);
|
||||
let runtime = test_runtime(driver.clone()).await;
|
||||
let mut sandbox = error_sandbox_record(
|
||||
"direct-timeout-id",
|
||||
"direct-timeout",
|
||||
"ImagePreparationTimedOut",
|
||||
);
|
||||
let mut operation = provisioning_deadline::new_record(0);
|
||||
operation.timeout_time = openshell_core::time::timestamp_from_millis(300_000).ok();
|
||||
operation.cleanup_completed_time = openshell_core::time::timestamp_from_millis(320_000).ok();
|
||||
sandbox.status.as_mut().unwrap().provisioning = Some(operation);
|
||||
runtime.store.put_message(&sandbox).await.unwrap();
|
||||
|
||||
let starting = runtime
|
||||
.start_sandbox("default", "direct-timeout")
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(starting.phase(), SandboxPhase::Starting as i32);
|
||||
assert_eq!(starting.object_id(), sandbox.object_id());
|
||||
assert!(driver.compute_exists.load(Ordering::SeqCst));
|
||||
assert_eq!(driver.start_calls(), 1);
|
||||
assert_eq!(driver.stop_calls(), 0);
|
||||
assert_eq!(driver.delete_calls(), 0);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn failed_error_stop_retains_retryable_state_without_claiming_success() {
|
||||
for observed_error in [false, true] {
|
||||
|
||||
@@ -0,0 +1,193 @@
|
||||
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
//! Baseline factory source fixture: no image-update trait or new hook required.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
|
||||
use openshell_core::proto::compute::v1::compute_driver_server::ComputeDriver;
|
||||
|
||||
use crate::test_support::FakeComputeDriver;
|
||||
use crate::{
|
||||
ComputeDriverBuildContext, ComputeDriverFactory, ComputeDriverInstance,
|
||||
ComputeDriverRegistration, ComputeDriverRegistry, DriverWatchStream, SharedComputeDriver,
|
||||
SharedComputeDriverConfigurationUpdates,
|
||||
};
|
||||
|
||||
// Written using only the baseline generated trait, not the new optional trait.
|
||||
type BaselineDriver =
|
||||
Arc<dyn ComputeDriver<WatchSandboxesStream = DriverWatchStream> + Send + Sync>;
|
||||
|
||||
struct BaselineFactory {
|
||||
driver: BaselineDriver,
|
||||
calls: Arc<AtomicUsize>,
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl ComputeDriverFactory for BaselineFactory {
|
||||
async fn build(
|
||||
&self,
|
||||
_context: ComputeDriverBuildContext<'_>,
|
||||
) -> openshell_core::Result<ComputeDriverInstance> {
|
||||
self.calls.fetch_add(1, Ordering::SeqCst);
|
||||
// This unchanged baseline expression must compile without a new trait.
|
||||
Ok(ComputeDriverInstance::InProcess(self.driver.clone()))
|
||||
}
|
||||
}
|
||||
|
||||
struct OptInFactory {
|
||||
driver: Arc<FakeComputeDriver>,
|
||||
legacy_build_calls: Arc<AtomicUsize>,
|
||||
opt_in_calls: Arc<AtomicUsize>,
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl ComputeDriverFactory for OptInFactory {
|
||||
async fn build(
|
||||
&self,
|
||||
_context: ComputeDriverBuildContext<'_>,
|
||||
) -> openshell_core::Result<ComputeDriverInstance> {
|
||||
self.legacy_build_calls.fetch_add(1, Ordering::SeqCst);
|
||||
Ok(ComputeDriverInstance::InProcess(self.driver.clone()))
|
||||
}
|
||||
|
||||
async fn build_with_configuration_updates(
|
||||
&self,
|
||||
_context: ComputeDriverBuildContext<'_>,
|
||||
) -> openshell_core::Result<(
|
||||
ComputeDriverInstance,
|
||||
Option<SharedComputeDriverConfigurationUpdates>,
|
||||
)> {
|
||||
self.opt_in_calls.fetch_add(1, Ordering::SeqCst);
|
||||
Ok((
|
||||
ComputeDriverInstance::InProcess(self.driver.clone()),
|
||||
Some(self.driver.clone()),
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
async fn registered_runtime(factory: impl ComputeDriverFactory + 'static) -> crate::ComputeRuntime {
|
||||
let mut registry = ComputeDriverRegistry::new();
|
||||
registry
|
||||
.install(ComputeDriverRegistration::new("fixture", 0, None, factory).unwrap())
|
||||
.unwrap();
|
||||
let selection = registry.select(Some("fixture")).unwrap();
|
||||
let config = openshell_core::Config::new(None);
|
||||
let overrides = BTreeMap::new();
|
||||
let startup = crate::compute::driver_config::DriverStartupContext {
|
||||
file: None,
|
||||
guest_tls: None,
|
||||
network_trust: None,
|
||||
network_trust_configured: false,
|
||||
gateway_port: 8080,
|
||||
gateway_tls_enabled: false,
|
||||
endpoint_overrides: &overrides,
|
||||
};
|
||||
let store = Arc::new(crate::persistence::test_store().await);
|
||||
crate::ensure_default_workspace(&store).await.unwrap();
|
||||
let mut sandbox = openshell_core::proto::Sandbox {
|
||||
metadata: Some(openshell_core::proto::ObjectMeta {
|
||||
id: "fixture-sandbox".into(),
|
||||
name: "fixture-sandbox".into(),
|
||||
workspace: "default".into(),
|
||||
..Default::default()
|
||||
}),
|
||||
spec: Some(openshell_core::proto::SandboxSpec {
|
||||
template: Some(openshell_core::proto::SandboxTemplate {
|
||||
image: "example/workload:v1".into(),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
sandbox.set_phase(openshell_core::proto::SandboxPhase::Stopped as i32);
|
||||
store.put_message(&sandbox).await.unwrap();
|
||||
|
||||
crate::build_compute_runtime(
|
||||
®istry,
|
||||
&selection,
|
||||
&config,
|
||||
startup,
|
||||
store,
|
||||
crate::sandbox_index::SandboxIndex::new(),
|
||||
crate::sandbox_watch::SandboxWatchBus::new(),
|
||||
crate::tracing_bus::TracingLogBus::new(),
|
||||
Arc::new(crate::supervisor_session::SupervisorSessionRegistry::new()),
|
||||
tokio::sync::watch::channel(false).1,
|
||||
)
|
||||
.await
|
||||
.unwrap()
|
||||
.with_sandbox_updates(true)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn baseline_factory_and_trait_object_still_build_without_opt_in() {
|
||||
// This concrete fixture implements only the baseline generated trait.
|
||||
let driver: BaselineDriver = Arc::new(crate::compute::NoopTestDriver::default());
|
||||
let shared: SharedComputeDriver = driver.clone();
|
||||
// Exhaustive baseline enum match also prevents adding a required new variant.
|
||||
match ComputeDriverInstance::InProcess(shared) {
|
||||
ComputeDriverInstance::InProcess(actual) => assert!(Arc::ptr_eq(&actual, &driver)),
|
||||
ComputeDriverInstance::ManagedRemote(_) => panic!("expected in-process driver"),
|
||||
}
|
||||
let calls = Arc::new(AtomicUsize::new(0));
|
||||
let runtime = registered_runtime(BaselineFactory {
|
||||
driver,
|
||||
calls: calls.clone(),
|
||||
})
|
||||
.await;
|
||||
assert_eq!(calls.load(Ordering::SeqCst), 1);
|
||||
assert_eq!(
|
||||
runtime
|
||||
.update_sandbox_image("fixture-sandbox", "example/workload:v2".into(), 1)
|
||||
.await
|
||||
.unwrap_err()
|
||||
.code(),
|
||||
tonic::Code::Unimplemented
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn capability_alone_does_not_register_an_in_process_image_updater() {
|
||||
let driver = Arc::new(FakeComputeDriver::new().with_sandbox_image_updates());
|
||||
let runtime = registered_runtime(BaselineFactory {
|
||||
driver: driver.clone(),
|
||||
calls: Arc::new(AtomicUsize::new(0)),
|
||||
})
|
||||
.await;
|
||||
driver.clear_calls();
|
||||
assert_eq!(
|
||||
runtime
|
||||
.update_sandbox_image("fixture-sandbox", "example/workload:v2".into(), 1)
|
||||
.await
|
||||
.unwrap_err()
|
||||
.code(),
|
||||
tonic::Code::Unimplemented
|
||||
);
|
||||
assert!(driver.calls().is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn registered_opt_in_hook_supplies_the_optional_updater() {
|
||||
let legacy_calls = Arc::new(AtomicUsize::new(0));
|
||||
let opt_in_calls = Arc::new(AtomicUsize::new(0));
|
||||
let runtime = registered_runtime(OptInFactory {
|
||||
driver: Arc::new(FakeComputeDriver::new().with_sandbox_image_updates()),
|
||||
legacy_build_calls: legacy_calls.clone(),
|
||||
opt_in_calls: opt_in_calls.clone(),
|
||||
})
|
||||
.await;
|
||||
assert_eq!(legacy_calls.load(Ordering::SeqCst), 0);
|
||||
assert_eq!(opt_in_calls.load(Ordering::SeqCst), 1);
|
||||
let updated = runtime
|
||||
.update_sandbox_image("fixture-sandbox", "example/workload:v2".into(), 1)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
updated.spec.unwrap().template.unwrap().image,
|
||||
"example/workload:v2"
|
||||
);
|
||||
}
|
||||
@@ -143,6 +143,9 @@ pub struct GatewayFileSection {
|
||||
/// Absolute preparation budget for new attempts; existing deadlines persist.
|
||||
#[serde(default)]
|
||||
pub image_preparation_timeout_seconds: Option<u32>,
|
||||
/// Explicit activation only after the compatible deployment rollout ends.
|
||||
#[serde(default)]
|
||||
pub enable_sandbox_updates: Option<bool>,
|
||||
#[serde(default)]
|
||||
pub grpc_rate_limit_requests: Option<u64>,
|
||||
#[serde(default)]
|
||||
|
||||
@@ -16,6 +16,9 @@ use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
/// Initial stopped-sandbox configuration update protocol advertised by gateways.
|
||||
pub const SANDBOX_UPDATE_PROTOCOL: u32 = 1;
|
||||
|
||||
const MEMBER_OBJECT_TYPE: &str = "gateway_member";
|
||||
|
||||
/// How long a membership record stays valid without a refresh.
|
||||
@@ -43,6 +46,9 @@ fn member_object_id(replica_id: &str) -> String {
|
||||
struct MemberPayload {
|
||||
replica_id: String,
|
||||
peer_endpoint: String,
|
||||
/// Binary capability, not feature activation. Missing denotes old replicas.
|
||||
#[serde(default)]
|
||||
sandbox_update_protocol: u32,
|
||||
}
|
||||
|
||||
/// A gateway replica that was alive as of its last refresh.
|
||||
@@ -71,6 +77,7 @@ impl GatewayMemberIndex {
|
||||
let payload = MemberPayload {
|
||||
replica_id: replica_id.to_string(),
|
||||
peer_endpoint: peer_endpoint.to_string(),
|
||||
sandbox_update_protocol: SANDBOX_UPDATE_PROTOCOL,
|
||||
};
|
||||
let bytes = serde_json::to_vec(&payload)
|
||||
.map_err(|err| PersistenceError::Encode(err.to_string()))?;
|
||||
|
||||
@@ -242,6 +242,16 @@ impl OpenShellService {
|
||||
// Trait impl — thin delegation to submodules
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl openshell_core::proto::sandbox_updates_server::SandboxUpdates for OpenShellService {
|
||||
async fn update_sandbox(
|
||||
&self,
|
||||
request: Request<openshell_core::proto::UpdateSandboxRequest>,
|
||||
) -> Result<Response<SandboxResponse>, Status> {
|
||||
mutation_replay::run(&self.state, request).await
|
||||
}
|
||||
}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl OpenShell for OpenShellService {
|
||||
async fn health(
|
||||
|
||||
@@ -102,6 +102,9 @@ struct AuthorizationBarrier {
|
||||
pub(super) trait Mutation: Message + Default + Send + Sync + 'static {
|
||||
type Output: Send + 'static;
|
||||
const METHOD: &'static str;
|
||||
/// Preserve legacy replay keys while allowing independent services to use
|
||||
/// fully qualified identities for newly introduced methods.
|
||||
const REPLAY_METHOD: &'static str = Self::METHOD;
|
||||
const PROTECTED: bool = false;
|
||||
// Streaming producers, not returned stream handles, confirm completion.
|
||||
const DEFERRED: bool = false;
|
||||
@@ -217,7 +220,7 @@ pub(super) async fn run<M: Mutation>(
|
||||
let bucket = format!("_mutation/{caller}");
|
||||
let key = hash_json(&serde_json::json!([
|
||||
caller,
|
||||
M::METHOD,
|
||||
M::REPLAY_METHOD,
|
||||
scope.name,
|
||||
request_id
|
||||
]))?;
|
||||
|
||||
@@ -335,10 +335,11 @@ async fn profile_scope(
|
||||
}
|
||||
|
||||
macro_rules! mutation {
|
||||
($req:ty, $resp:ty, $method:literal, $handler:path, $auth:expr, $capture:expr, $restore:expr) => {
|
||||
($req:ty, $resp:ty, $method:literal, $handler:path, $auth:expr, $capture:expr, $restore:expr $(, $replay_method:literal)?) => {
|
||||
impl Mutation for $req {
|
||||
type Output = $resp;
|
||||
const METHOD: &'static str = $method;
|
||||
$(const REPLAY_METHOD: &'static str = $replay_method;)?
|
||||
const PROTECTED: bool = true;
|
||||
fn request_id(&self) -> &str {
|
||||
&self.request_id
|
||||
@@ -389,7 +390,7 @@ macro_rules! scoped_mutation {
|
||||
}
|
||||
|
||||
macro_rules! sandbox_scoped_mutation {
|
||||
($req:ty, $resp:ty, $method:literal, $handler:path, $role:ident, $capture:expr, $restore:expr) => {
|
||||
($req:ty, $resp:ty, $method:literal, $handler:path, $role:ident, $capture:expr, $restore:expr $(, $replay_method:literal)?) => {
|
||||
mutation!(
|
||||
$req,
|
||||
$resp,
|
||||
@@ -406,6 +407,7 @@ macro_rules! sandbox_scoped_mutation {
|
||||
},
|
||||
$capture,
|
||||
$restore
|
||||
$(, $replay_method)?
|
||||
);
|
||||
};
|
||||
}
|
||||
@@ -423,7 +425,7 @@ fn sandbox_receipt(
|
||||
}
|
||||
|
||||
macro_rules! sandbox_mutation {
|
||||
($req:ty, $method:literal, $handler:path) => {
|
||||
($req:ty, $method:literal, $handler:path $(, $replay_method:literal)?) => {
|
||||
sandbox_scoped_mutation!(
|
||||
$req,
|
||||
SandboxResponse,
|
||||
@@ -444,6 +446,7 @@ macro_rules! sandbox_mutation {
|
||||
service_urls: HashMap::new(),
|
||||
})
|
||||
}
|
||||
$(, $replay_method)?
|
||||
);
|
||||
};
|
||||
}
|
||||
@@ -471,6 +474,12 @@ scoped_mutation!(
|
||||
})
|
||||
}
|
||||
);
|
||||
sandbox_mutation!(
|
||||
openshell_core::proto::UpdateSandboxRequest,
|
||||
"UpdateSandbox",
|
||||
sandbox::handle_update_sandbox,
|
||||
"/openshell.v1.SandboxUpdates/UpdateSandbox"
|
||||
);
|
||||
sandbox_mutation!(
|
||||
StartSandboxRequest,
|
||||
"StartSandbox",
|
||||
|
||||
@@ -315,6 +315,102 @@ pub(super) async fn handle_create_sandbox(
|
||||
result
|
||||
}
|
||||
|
||||
/// Only the workload image is mutable in the first update contract. Runtime
|
||||
/// replacement is an explicit Stop/Update/Start workflow, never Delete/Create.
|
||||
pub(super) async fn handle_update_sandbox(
|
||||
state: &Arc<ServerState>,
|
||||
request: Request<openshell_core::proto::UpdateSandboxRequest>,
|
||||
) -> Result<Response<SandboxResponse>, Status> {
|
||||
let principal = super::extract_principal(&request)?;
|
||||
let request = request.into_inner();
|
||||
validate_image_update_request(&request)?;
|
||||
let sandbox = resolve_and_authorize_sandbox_name(
|
||||
state,
|
||||
&principal,
|
||||
&request.name,
|
||||
crate::auth::workspace_authz::selected_workspace_name(request.workspace_scope.as_ref())?,
|
||||
MinWorkspaceRole::User,
|
||||
)
|
||||
.await?;
|
||||
super::workspace::resolve_workspace(state.store.as_ref(), sandbox.object_workspace())
|
||||
.await?
|
||||
.ensure_active()?;
|
||||
let mut candidate = sandbox.clone();
|
||||
let spec = candidate
|
||||
.spec
|
||||
.as_mut()
|
||||
.ok_or_else(|| Status::failed_precondition("sandbox has no spec"))?;
|
||||
if let Some(image) = &request.image {
|
||||
spec.template
|
||||
.get_or_insert_with(Default::default)
|
||||
.image
|
||||
.clone_from(image);
|
||||
}
|
||||
openshell_core::sandbox_update::apply_configuration(
|
||||
&mut candidate,
|
||||
request.command.as_ref(),
|
||||
request.environment.as_ref(),
|
||||
request.labels.as_ref(),
|
||||
request.annotations.as_ref(),
|
||||
);
|
||||
let candidate_spec = candidate.spec.as_ref().expect("spec checked above");
|
||||
validate_sandbox_spec(&request.name, candidate_spec)?;
|
||||
|
||||
let sandbox = Box::pin(
|
||||
state
|
||||
.compute
|
||||
.update_sandbox_configuration(sandbox.object_id(), &request),
|
||||
)
|
||||
.await?;
|
||||
Ok(Response::new(SandboxResponse {
|
||||
sandbox: Some(sandbox),
|
||||
service_urls: HashMap::new(),
|
||||
}))
|
||||
}
|
||||
|
||||
fn validate_image_update_request(
|
||||
request: &openshell_core::proto::UpdateSandboxRequest,
|
||||
) -> Result<(), Status> {
|
||||
if request.name.is_empty() {
|
||||
return Err(Status::invalid_argument("name is required"));
|
||||
}
|
||||
if request.expected_resource_version == 0 {
|
||||
return Err(Status::invalid_argument(
|
||||
"expected_resource_version must be nonzero",
|
||||
));
|
||||
}
|
||||
if let Some(image) = &request.image
|
||||
&& (image.is_empty()
|
||||
|| image.len() > super::MAX_TEMPLATE_STRING_LEN
|
||||
|| image.chars().any(|c| c.is_whitespace() || c.is_control()))
|
||||
{
|
||||
return Err(Status::invalid_argument(
|
||||
"image must be a nonempty bounded OCI reference without whitespace or control characters",
|
||||
));
|
||||
}
|
||||
for map in [request.labels.as_ref(), request.annotations.as_ref()]
|
||||
.into_iter()
|
||||
.flatten()
|
||||
{
|
||||
if map
|
||||
.values
|
||||
.keys()
|
||||
.any(|key| openshell_core::sandbox_update::is_system_metadata_key(key))
|
||||
{
|
||||
return Err(Status::invalid_argument(
|
||||
"system metadata cannot be updated",
|
||||
));
|
||||
}
|
||||
}
|
||||
if let Some(labels) = &request.labels {
|
||||
super::validation::validate_labels(&labels.values)?;
|
||||
}
|
||||
if let Some(annotations) = &request.annotations {
|
||||
super::validation::validate_annotations(&annotations.values, "annotations")?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Allocate a gateway-owned staging slot for a local rootfs tar archive.
|
||||
///
|
||||
/// The caller writes the archive to the returned path and then names the token
|
||||
@@ -786,10 +882,7 @@ fn validate_create_sandbox_request_pre_io(
|
||||
workload_template_name: &str,
|
||||
) -> Result<(), Status> {
|
||||
// Validate labels (keys and values must meet Kubernetes requirements).
|
||||
for (key, value) in &request.labels {
|
||||
crate::grpc::validation::validate_label_key(key)?;
|
||||
crate::grpc::validation::validate_label_value(value)?;
|
||||
}
|
||||
crate::grpc::validation::validate_labels(&request.labels)?;
|
||||
crate::grpc::validation::validate_caller_annotations(&request.annotations, "annotations")?;
|
||||
|
||||
if request.service_exposures.len() > MAX_CREATE_SERVICE_EXPOSURES {
|
||||
@@ -7999,6 +8092,21 @@ mod tests {
|
||||
assert_eq!(process.run_as_group, "1234");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn create_sandbox_accepts_more_than_128_labels() {
|
||||
let request = CreateSandboxRequest {
|
||||
name: "many-labels".into(),
|
||||
spec: Some(SandboxSpec::default()),
|
||||
labels: (0..129)
|
||||
.map(|index| (format!("label-{index}"), "value".into()))
|
||||
.collect(),
|
||||
..Default::default()
|
||||
};
|
||||
for workload_template in ["", "workload-template"] {
|
||||
validate_create_sandbox_request_pre_io(&request, workload_template).unwrap();
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn create_sandbox_still_rejects_long_label_values() {
|
||||
let state = test_server_state().await;
|
||||
@@ -10470,3 +10578,6 @@ mod tests {
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod image_update_tests;
|
||||
|
||||
@@ -0,0 +1,459 @@
|
||||
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
use super::*;
|
||||
use crate::auth::audit::ConnectionSource;
|
||||
use crate::grpc::test_support::{authed_request, test_server_state};
|
||||
use crate::test_support::{FakeComputeDriver, FakeComputeDriverCall};
|
||||
use openshell_core::proto::compute::v1::GetCapabilitiesResponse;
|
||||
use openshell_core::proto::datamodel::v1::ObjectMeta;
|
||||
use openshell_core::proto::sandbox_updates_client::SandboxUpdatesClient;
|
||||
use openshell_core::proto::{SandboxCommandUpdate, UpdateSandboxRequest};
|
||||
use tonic::Code;
|
||||
|
||||
struct RunningService(tokio::task::JoinHandle<()>);
|
||||
|
||||
impl Drop for RunningService {
|
||||
fn drop(&mut self) {
|
||||
self.0.abort();
|
||||
}
|
||||
}
|
||||
|
||||
async fn wire_client(
|
||||
state: Arc<ServerState>,
|
||||
) -> (
|
||||
SandboxUpdatesClient<tonic::transport::Channel>,
|
||||
RunningService,
|
||||
) {
|
||||
use crate::multiplex::MultiplexService;
|
||||
use openshell_core::proto::open_shell_client::OpenShellClient;
|
||||
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let address = listener.local_addr().unwrap();
|
||||
// Use the production gateway multiplexer, not a standalone tonic server:
|
||||
// this proves registration, routing and principal injection on the same
|
||||
// listener as the unchanged OpenShell service.
|
||||
let server = RunningService(tokio::spawn(async move {
|
||||
let mut connections = tokio::task::JoinSet::new();
|
||||
loop {
|
||||
let (stream, peer) = listener.accept().await.unwrap();
|
||||
let service = MultiplexService::new(state.clone());
|
||||
connections.spawn(async move {
|
||||
service
|
||||
.serve(stream, ConnectionSource::Peer(peer))
|
||||
.await
|
||||
.unwrap();
|
||||
});
|
||||
}
|
||||
}));
|
||||
let channel = tonic::transport::Endpoint::from_shared(format!("http://{address}"))
|
||||
.unwrap()
|
||||
.connect()
|
||||
.await
|
||||
.unwrap();
|
||||
OpenShellClient::new(channel.clone())
|
||||
.health(openshell_core::proto::HealthRequest {})
|
||||
.await
|
||||
.unwrap();
|
||||
(SandboxUpdatesClient::new(channel), server)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn image_updates_have_their_own_authenticated_workspace_scoped_service() {
|
||||
use crate::auth::method_authz::{AuthMode, lookup};
|
||||
use crate::grpc::mutation_replay::Mutation;
|
||||
|
||||
let path = "/openshell.v1.SandboxUpdates/UpdateSandbox";
|
||||
let authorization = lookup(path).expect("new service must participate in gateway auth");
|
||||
assert_eq!(authorization.auth_mode, AuthMode::Bearer);
|
||||
assert_eq!(authorization.scope.as_deref(), Some("sandbox:write"));
|
||||
assert_eq!(authorization.workspace_role.as_deref(), Some("user"));
|
||||
assert!(lookup("/openshell.v1.OpenShell/UpdateSandbox").is_none());
|
||||
assert_eq!(<UpdateSandboxRequest as Mutation>::REPLAY_METHOD, path);
|
||||
}
|
||||
|
||||
fn request() -> UpdateSandboxRequest {
|
||||
UpdateSandboxRequest {
|
||||
workspace_scope: Some(openshell_core::proto::workspace_selector("default")),
|
||||
name: "image-update".into(),
|
||||
image: Some("example/workload:new".into()),
|
||||
expected_resource_version: 1,
|
||||
request_id: String::new(),
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn image_only_request_requires_target_image_and_conditional_version() {
|
||||
let valid = request();
|
||||
validate_image_update_request(&valid).unwrap();
|
||||
for image in ["", " ", "example/workload:tag\n", "bad image", "a\0b"] {
|
||||
assert_eq!(
|
||||
validate_image_update_request(&UpdateSandboxRequest {
|
||||
image: Some(image.into()),
|
||||
..valid.clone()
|
||||
})
|
||||
.unwrap_err()
|
||||
.code(),
|
||||
Code::InvalidArgument
|
||||
);
|
||||
}
|
||||
assert!(
|
||||
validate_image_update_request(&UpdateSandboxRequest {
|
||||
image: Some("a".repeat(super::super::MAX_TEMPLATE_STRING_LEN + 1)),
|
||||
..valid.clone()
|
||||
})
|
||||
.is_err()
|
||||
);
|
||||
assert!(
|
||||
validate_image_update_request(&UpdateSandboxRequest {
|
||||
name: String::new(),
|
||||
..valid.clone()
|
||||
})
|
||||
.is_err()
|
||||
);
|
||||
assert!(
|
||||
validate_image_update_request(&UpdateSandboxRequest {
|
||||
expected_resource_version: 0,
|
||||
..valid
|
||||
})
|
||||
.is_err()
|
||||
);
|
||||
}
|
||||
|
||||
async fn state() -> (Arc<ServerState>, Arc<FakeComputeDriver>, Sandbox) {
|
||||
let mut state = test_server_state().await;
|
||||
Arc::get_mut(&mut state)
|
||||
.unwrap()
|
||||
.config
|
||||
.enable_sandbox_updates = true;
|
||||
let driver = Arc::new(
|
||||
FakeComputeDriver::new().with_capabilities(GetCapabilitiesResponse {
|
||||
driver_name: "image-test".into(),
|
||||
default_image: "example/workload:old".into(),
|
||||
extension: Some(openshell_core::extension_protocol::extension_metadata(
|
||||
openshell_core::extension_protocol::ExtensionFamily::Compute,
|
||||
"openshell/image-test",
|
||||
"test",
|
||||
[openshell_core::extension_protocol::COMPUTE_UPDATE_SANDBOX_CONFIGURATION.into()],
|
||||
)),
|
||||
..Default::default()
|
||||
}),
|
||||
);
|
||||
let compute = crate::compute::ComputeRuntime::from_driver(
|
||||
"image-test".into(),
|
||||
driver.clone(),
|
||||
None,
|
||||
state.store.clone(),
|
||||
crate::sandbox_index::SandboxIndex::new(),
|
||||
crate::sandbox_watch::SandboxWatchBus::new(),
|
||||
crate::tracing_bus::TracingLogBus::new(),
|
||||
Arc::new(crate::supervisor_session::SupervisorSessionRegistry::new()),
|
||||
)
|
||||
.await
|
||||
.unwrap()
|
||||
.with_image_update_driver(Some(driver.clone()))
|
||||
.with_sandbox_updates(true);
|
||||
Arc::get_mut(&mut state).unwrap().compute = compute;
|
||||
let mut sandbox = Sandbox {
|
||||
metadata: Some(ObjectMeta {
|
||||
id: "stable-sandbox-id".into(),
|
||||
name: "image-update".into(),
|
||||
workspace: "default".into(),
|
||||
..Default::default()
|
||||
}),
|
||||
spec: Some(SandboxSpec {
|
||||
template: Some(SandboxTemplate {
|
||||
image: "example/workload:old".into(),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
sandbox.set_phase(SandboxPhase::Stopped as i32);
|
||||
state.store.put_message(&sandbox).await.unwrap();
|
||||
let sandbox = state
|
||||
.store
|
||||
.get_message::<Sandbox>(sandbox.object_id())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
(state, driver, sandbox)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn public_image_update_checks_workspace_and_missing_resource() {
|
||||
let (state, driver, sandbox) = state().await;
|
||||
let mut req = request();
|
||||
req.expected_resource_version = sandbox.get_resource_version();
|
||||
req.workspace_scope = None;
|
||||
assert_eq!(
|
||||
handle_update_sandbox(&state, authed_request(req.clone()))
|
||||
.await
|
||||
.unwrap_err()
|
||||
.code(),
|
||||
Code::InvalidArgument
|
||||
);
|
||||
req.workspace_scope = Some(openshell_core::proto::workspace_selector("default"));
|
||||
req.name = "missing-sandbox".into();
|
||||
assert_eq!(
|
||||
handle_update_sandbox(&state, authed_request(req))
|
||||
.await
|
||||
.unwrap_err()
|
||||
.code(),
|
||||
Code::NotFound
|
||||
);
|
||||
assert!(
|
||||
!driver
|
||||
.calls()
|
||||
.iter()
|
||||
.any(|call| matches!(call, FakeComputeDriverCall::UpdateSandboxImage { .. }))
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn public_image_update_replays_without_reapplying_or_touching_same_name_replacement() {
|
||||
let (mut state, driver, sandbox) = state().await;
|
||||
// Exercise production routing with the explicit local developer identity.
|
||||
Arc::get_mut(&mut state)
|
||||
.unwrap()
|
||||
.config
|
||||
.auth
|
||||
.allow_unauthenticated_users = true;
|
||||
let directory = tempfile::tempdir().unwrap();
|
||||
let key = directory.path().join("private-key");
|
||||
std::fs::write(&key, b"test-only stable private fingerprint material").unwrap();
|
||||
Arc::get_mut(&mut state).unwrap().config.gateway_jwt =
|
||||
Some(openshell_core::config::GatewayJwtConfig {
|
||||
signing_key_path: key,
|
||||
public_key_path: directory.path().join("public"),
|
||||
kid_path: directory.path().join("kid"),
|
||||
gateway_id: "test".into(),
|
||||
ttl_secs: None,
|
||||
});
|
||||
let (mut client, _server) = wire_client(state.clone()).await;
|
||||
let req = UpdateSandboxRequest {
|
||||
expected_resource_version: sandbox.get_resource_version(),
|
||||
request_id: uuid::Uuid::new_v4().to_string(),
|
||||
command: Some(SandboxCommandUpdate {
|
||||
argv: vec!["/usr/local/bin/start-agent".into(), "--updated".into()],
|
||||
}),
|
||||
environment: Some(openshell_core::proto::SandboxMapUpdate {
|
||||
values: HashMap::from([("AGENT_MODE".into(), "updated".into())]),
|
||||
}),
|
||||
labels: Some(openshell_core::proto::SandboxMapUpdate {
|
||||
values: HashMap::from([("team".into(), "agents".into())]),
|
||||
}),
|
||||
annotations: Some(openshell_core::proto::SandboxMapUpdate {
|
||||
values: HashMap::from([("example.com/revision".into(), "2".into())]),
|
||||
}),
|
||||
..request()
|
||||
};
|
||||
let updated = client
|
||||
.update_sandbox(req.clone())
|
||||
.await
|
||||
.unwrap()
|
||||
.into_inner()
|
||||
.sandbox
|
||||
.unwrap();
|
||||
assert_eq!(updated.object_id(), sandbox.object_id());
|
||||
assert_eq!(
|
||||
updated
|
||||
.spec
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.template
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.image,
|
||||
req.image.as_ref().unwrap().as_str()
|
||||
);
|
||||
assert_eq!(
|
||||
updated.spec.as_ref().unwrap().command,
|
||||
req.command.as_ref().unwrap().argv
|
||||
);
|
||||
assert_eq!(
|
||||
updated.spec.as_ref().unwrap().environment,
|
||||
req.environment.as_ref().unwrap().values
|
||||
);
|
||||
assert_eq!(
|
||||
updated.metadata.as_ref().unwrap().labels,
|
||||
req.labels.as_ref().unwrap().values
|
||||
);
|
||||
let replay = client.update_sandbox(req.clone()).await.unwrap();
|
||||
assert_eq!(replay.metadata().get("openshell-replayed").unwrap(), "true");
|
||||
assert_eq!(replay.get_ref().sandbox.as_ref().unwrap(), &updated);
|
||||
assert_eq!(driver.calls().iter().filter(|call| matches!(call, FakeComputeDriverCall::UpdateSandboxConfiguration { request } if !request.preconditions.as_ref().unwrap().validate_only)).count(), 1);
|
||||
let changed = UpdateSandboxRequest {
|
||||
image: Some("example/workload:other".into()),
|
||||
..req.clone()
|
||||
};
|
||||
assert_eq!(
|
||||
client.update_sandbox(changed).await.unwrap_err().code(),
|
||||
Code::FailedPrecondition
|
||||
);
|
||||
state
|
||||
.store
|
||||
.delete(Sandbox::object_type(), sandbox.object_id())
|
||||
.await
|
||||
.unwrap();
|
||||
let mut replacement = sandbox.clone();
|
||||
let metadata = replacement.metadata.as_mut().unwrap();
|
||||
metadata.id = "replacement-sandbox-id".into();
|
||||
metadata.resource_version = 0;
|
||||
state.store.put_message(&replacement).await.unwrap();
|
||||
assert_eq!(
|
||||
client.update_sandbox(req).await.unwrap_err().code(),
|
||||
Code::FailedPrecondition
|
||||
);
|
||||
let replacement = state
|
||||
.store
|
||||
.get_message::<Sandbox>(replacement.object_id())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert_eq!(replacement.spec, sandbox.spec);
|
||||
assert_eq!(driver.calls().iter().filter(|call| matches!(call, FakeComputeDriverCall::UpdateSandboxConfiguration { request } if !request.preconditions.as_ref().unwrap().validate_only)).count(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn configuration_update_accepts_more_than_128_labels() {
|
||||
let update = UpdateSandboxRequest {
|
||||
labels: Some(openshell_core::proto::SandboxMapUpdate {
|
||||
values: (0..129)
|
||||
.map(|index| (format!("label-{index}"), "value".into()))
|
||||
.collect(),
|
||||
}),
|
||||
..request()
|
||||
};
|
||||
validate_image_update_request(&update).unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn configuration_update_presence_and_reserved_metadata() {
|
||||
let mut update = request();
|
||||
update.image = None;
|
||||
assert!(validate_image_update_request(&update).is_ok());
|
||||
update.command = Some(SandboxCommandUpdate::default());
|
||||
validate_image_update_request(&update).unwrap();
|
||||
// User metadata under these domains was accepted at creation and remains
|
||||
// editable; only explicit gateway identity and lifecycle keys are reserved.
|
||||
for key in ["openshell.ai/user-note", "agents.x-k8s.io/user-note"] {
|
||||
update.annotations = Some(openshell_core::proto::SandboxMapUpdate {
|
||||
values: HashMap::from([(key.into(), "new".into())]),
|
||||
});
|
||||
validate_image_update_request(&update).unwrap();
|
||||
}
|
||||
update.annotations = None;
|
||||
for key in [
|
||||
"internal.openshell.ai/auth-epoch",
|
||||
"openshell.ai/sandbox-id",
|
||||
"openshell.nvidia.com/retention",
|
||||
] {
|
||||
update.annotations = Some(openshell_core::proto::SandboxMapUpdate {
|
||||
values: HashMap::from([(key.into(), "forged".into())]),
|
||||
});
|
||||
assert!(validate_image_update_request(&update).is_err());
|
||||
update.labels = update.annotations.take();
|
||||
assert!(validate_image_update_request(&update).is_err());
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn pending_update_retry_preserves_later_update_config_annotations() {
|
||||
use openshell_core::proto::{SandboxMapUpdate, UpdateConfigRequest};
|
||||
|
||||
let (state, driver, sandbox) = state().await;
|
||||
let mut update = UpdateSandboxRequest {
|
||||
expected_resource_version: sandbox.get_resource_version(),
|
||||
annotations: Some(SandboxMapUpdate {
|
||||
values: HashMap::from([
|
||||
("example.com/revision".into(), "original".into()),
|
||||
("example.com/retained".into(), "value".into()),
|
||||
]),
|
||||
}),
|
||||
..request()
|
||||
};
|
||||
driver.set_image_update_error(Some(Code::Unavailable));
|
||||
assert_eq!(
|
||||
handle_update_sandbox(&state, authed_request(update.clone()))
|
||||
.await
|
||||
.unwrap_err()
|
||||
.code(),
|
||||
Code::Unavailable
|
||||
);
|
||||
let pending = state
|
||||
.store
|
||||
.get_message::<Sandbox>(sandbox.object_id())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
|
||||
// Exercise the existing public annotation writer while the update is
|
||||
// pending, including both a new key and an overwrite of a selected key.
|
||||
let annotations = crate::grpc::policy::handle_update_config(
|
||||
&state,
|
||||
authed_request(UpdateConfigRequest {
|
||||
sandbox: sandbox.object_name().into(),
|
||||
workspace_scope: update.workspace_scope.clone(),
|
||||
expected_resource_version: pending.get_resource_version(),
|
||||
setting_key: "log_level".into(),
|
||||
delete_setting: true,
|
||||
annotations: HashMap::from([
|
||||
("example.com/revision".into(), "later".into()),
|
||||
("example.com/added".into(), "new".into()),
|
||||
]),
|
||||
..Default::default()
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap()
|
||||
.into_inner()
|
||||
.annotations;
|
||||
assert_eq!(annotations.get("example.com/retained").unwrap(), "value");
|
||||
let current = state
|
||||
.store
|
||||
.get_message::<Sandbox>(sandbox.object_id())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
// Resume from stored intent, without knowing the original selection mask.
|
||||
update = UpdateSandboxRequest {
|
||||
workspace_scope: update.workspace_scope,
|
||||
name: update.name,
|
||||
expected_resource_version: current.get_resource_version(),
|
||||
..Default::default()
|
||||
};
|
||||
driver.set_image_update_error(None);
|
||||
let completed = handle_update_sandbox(&state, authed_request(update))
|
||||
.await
|
||||
.unwrap()
|
||||
.into_inner()
|
||||
.sandbox
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
completed.metadata.as_ref().unwrap().annotations,
|
||||
annotations
|
||||
);
|
||||
assert_eq!(completed.phase(), SandboxPhase::Stopped as i32);
|
||||
assert!(
|
||||
completed
|
||||
.status
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.provisioning
|
||||
.as_ref()
|
||||
.is_none_or(|record| !record.driver_operation_pending)
|
||||
);
|
||||
assert!(
|
||||
state
|
||||
.store
|
||||
.get_sandbox_image_state(sandbox.object_id())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap()
|
||||
.pending
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
@@ -459,6 +459,17 @@ pub(super) fn validate_string_map(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Validate label keys and values using Kubernetes syntax.
|
||||
pub(super) fn validate_labels(
|
||||
labels: &std::collections::HashMap<String, String>,
|
||||
) -> Result<(), Status> {
|
||||
for (key, value) in labels {
|
||||
validate_label_key(key)?;
|
||||
validate_label_value(value)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Validate object annotations.
|
||||
///
|
||||
/// Annotation keys use the same qualified-key shape as labels. Annotation
|
||||
|
||||
@@ -482,6 +482,7 @@ impl ServerState {
|
||||
credentials: credentials::CredentialRuntime,
|
||||
) -> Self {
|
||||
let replica_id = compute::lease::replica_id();
|
||||
let compute = compute.with_sandbox_updates(config.enable_sandbox_updates);
|
||||
compute.configure_ssh_identities(credentials.clone());
|
||||
let service_route_index = Arc::new(service_routing::ServiceRouteIndex::new(
|
||||
config.service_routing_discriminator().map(str::to_string),
|
||||
@@ -1388,6 +1389,7 @@ async fn terminate_signal() {
|
||||
|
||||
pub use compute::{
|
||||
AcquiredRemoteDriverEndpoint, DriverWatchStream, ManagedDriverProcess, SharedComputeDriver,
|
||||
SharedComputeDriverConfigurationUpdates,
|
||||
};
|
||||
|
||||
/// Driver instance returned by a compiled compute-driver factory.
|
||||
@@ -1463,6 +1465,23 @@ pub trait ComputeDriverFactory: Send + Sync {
|
||||
}
|
||||
|
||||
async fn build(&self, context: ComputeDriverBuildContext<'_>) -> Result<ComputeDriverInstance>;
|
||||
|
||||
/// Build with a separately registered, optional in-process image updater.
|
||||
///
|
||||
/// Existing factories only implement `build`; their baseline driver object
|
||||
/// remains sufficient and this hook defaults to no configuration-update support.
|
||||
/// Supporting factories return the same driver instance in both slots and
|
||||
/// advertise the negotiated configuration-update capability. Remote drivers expose
|
||||
/// their extension over the existing endpoint instead of using this hook.
|
||||
async fn build_with_configuration_updates(
|
||||
&self,
|
||||
context: ComputeDriverBuildContext<'_>,
|
||||
) -> Result<(
|
||||
ComputeDriverInstance,
|
||||
Option<SharedComputeDriverConfigurationUpdates>,
|
||||
)> {
|
||||
self.build(context).await.map(|instance| (instance, None))
|
||||
}
|
||||
}
|
||||
|
||||
/// One named compiled-driver registration.
|
||||
@@ -1913,7 +1932,10 @@ async fn build_compute_runtime(
|
||||
supervisor_recovery_timeout_secs: supervisor_pool
|
||||
.and_then(|pool| pool.recovery_timeout_secs),
|
||||
};
|
||||
let instance = registration.factory.build(build_context).await?;
|
||||
let (instance, image_updates) = registration
|
||||
.factory
|
||||
.build_with_configuration_updates(build_context)
|
||||
.await?;
|
||||
let supervisor_host = shared_supervision
|
||||
.into_inner()
|
||||
.map(|supervision| supervision.controller().clone());
|
||||
@@ -1935,7 +1957,8 @@ async fn build_compute_runtime(
|
||||
.map_err(|error| {
|
||||
Error::execution(format!("failed to create compute runtime: {error}"))
|
||||
})?
|
||||
.with_supervisor_host(supervisor_host),
|
||||
.with_supervisor_host(supervisor_host)
|
||||
.with_image_update_driver(image_updates),
|
||||
ComputeDriverInstance::ManagedRemote(_) if supervisor_host.is_some() => {
|
||||
return Err(Error::config(
|
||||
"only an in-process compute driver can use the gateway's shared supervision",
|
||||
@@ -2210,6 +2233,9 @@ pub(crate) async fn ensure_default_workspace(store: &Store) -> Result<()> {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod compute_factory_compatibility;
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{
|
||||
@@ -2701,7 +2727,7 @@ mod tests {
|
||||
.iter()
|
||||
.map(|service| service.name.as_str())
|
||||
.collect::<Vec<_>>(),
|
||||
["openshell.v1.OpenShell"]
|
||||
["openshell.v1.OpenShell", "openshell.v1.SandboxUpdates"]
|
||||
);
|
||||
|
||||
stop_listener(shutdown, handle).await;
|
||||
|
||||
@@ -18,6 +18,7 @@ use hyper_util::{
|
||||
};
|
||||
use metrics::{counter, histogram};
|
||||
use openshell_core::proto::open_shell_server::OpenShellServer;
|
||||
use openshell_core::proto::sandbox_updates_server::SandboxUpdatesServer;
|
||||
use openshell_core::proto::supervisor_host::v1::supervisor_host_server::SupervisorHostServer;
|
||||
use openshell_core::{
|
||||
Config,
|
||||
@@ -305,6 +306,13 @@ impl MultiplexService {
|
||||
openshell,
|
||||
);
|
||||
let reflection = self.state.reflection_service.clone();
|
||||
let sandbox_updates = SandboxUpdatesServer::new(OpenShellService::new(self.state.clone()))
|
||||
.max_decoding_message_size(MAX_GRPC_DECODE_SIZE);
|
||||
let sandbox_updates = GatewayInterceptorGrpcService::new(
|
||||
sandbox_updates,
|
||||
self.state.gateway_interceptors.clone(),
|
||||
Some(self.state.clone()),
|
||||
);
|
||||
let authz_policy = self.state.config.oidc.as_ref().map(|oidc| AuthzPolicy {
|
||||
admin_role: oidc.admin_role.clone(),
|
||||
user_role: oidc.user_role.clone(),
|
||||
@@ -312,7 +320,7 @@ impl MultiplexService {
|
||||
});
|
||||
let authenticator_chain = build_authenticator_chain(&self.state);
|
||||
let grpc_service = AuthGrpcRouter::with_peer_identity(
|
||||
GrpcRouter::new(openshell, reflection),
|
||||
GrpcRouter::new(openshell, reflection).with_sandbox_updates(sandbox_updates),
|
||||
authenticator_chain,
|
||||
authz_policy,
|
||||
self.state
|
||||
@@ -987,11 +995,12 @@ where
|
||||
}
|
||||
}
|
||||
|
||||
/// Combined gRPC service that routes between `OpenShell` and reflection.
|
||||
/// Combined gRPC service for public gateway services and reflection.
|
||||
#[derive(Clone)]
|
||||
pub struct GrpcRouter<N, R> {
|
||||
pub struct GrpcRouter<N, R, U = N> {
|
||||
openshell: N,
|
||||
reflection: R,
|
||||
sandbox_updates: Option<U>,
|
||||
}
|
||||
|
||||
impl<N, R> GrpcRouter<N, R> {
|
||||
@@ -999,13 +1008,22 @@ impl<N, R> GrpcRouter<N, R> {
|
||||
Self {
|
||||
openshell,
|
||||
reflection,
|
||||
sandbox_updates: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn with_sandbox_updates<U>(self, sandbox_updates: U) -> GrpcRouter<N, R, U> {
|
||||
GrpcRouter {
|
||||
openshell: self.openshell,
|
||||
reflection: self.reflection,
|
||||
sandbox_updates: Some(sandbox_updates),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub const REFLECTION_PATH_PREFIX: &str = "/grpc.reflection.v1.";
|
||||
|
||||
impl<N, R, B> tower::Service<Request<B>> for GrpcRouter<N, R>
|
||||
impl<N, R, U, B> tower::Service<Request<B>> for GrpcRouter<N, R, U>
|
||||
where
|
||||
N: tower::Service<Request<B>> + Clone + Send + 'static,
|
||||
N::Response: Send,
|
||||
@@ -1016,6 +1034,11 @@ where
|
||||
+ Send
|
||||
+ 'static,
|
||||
R::Future: Send,
|
||||
U: tower::Service<Request<B>, Response = N::Response, Error = N::Error>
|
||||
+ Clone
|
||||
+ Send
|
||||
+ 'static,
|
||||
U::Future: Send,
|
||||
B: Send + 'static,
|
||||
{
|
||||
type Response = N::Response;
|
||||
@@ -1030,6 +1053,13 @@ where
|
||||
if req.uri().path().starts_with(REFLECTION_PATH_PREFIX) {
|
||||
let mut svc = self.reflection.clone();
|
||||
Box::pin(async move { svc.ready().await?.call(req).await })
|
||||
} else if req
|
||||
.uri()
|
||||
.path()
|
||||
.starts_with("/openshell.v1.SandboxUpdates/")
|
||||
&& let Some(mut svc) = self.sandbox_updates.clone()
|
||||
{
|
||||
Box::pin(async move { svc.ready().await?.call(req).await })
|
||||
} else {
|
||||
let mut svc = self.openshell.clone();
|
||||
Box::pin(async move { svc.ready().await?.call(req).await })
|
||||
@@ -2836,6 +2866,13 @@ mod tests {
|
||||
otel_span_name(&http::Method::POST, "/openshell.v1.OpenShell/CreateSandbox"),
|
||||
"openshell.v1.OpenShell/CreateSandbox"
|
||||
);
|
||||
assert_eq!(
|
||||
otel_span_name(
|
||||
&http::Method::POST,
|
||||
"/openshell.v1.SandboxUpdates/UpdateSandbox"
|
||||
),
|
||||
"openshell.v1.SandboxUpdates/UpdateSandbox"
|
||||
);
|
||||
}
|
||||
|
||||
/// Non-RPC paths use a low-cardinality method-only name because sandbox
|
||||
@@ -2905,10 +2942,12 @@ mod tests {
|
||||
name,
|
||||
calls: calls.clone(),
|
||||
};
|
||||
let mut router = GrpcRouter::new(service("openshell"), service("reflection"));
|
||||
let mut router = GrpcRouter::new(service("openshell"), service("reflection"))
|
||||
.with_sandbox_updates(service("sandbox_updates"));
|
||||
|
||||
for path in [
|
||||
"/openshell.v1.OpenShell/Health",
|
||||
"/openshell.v1.SandboxUpdates/UpdateSandbox",
|
||||
"/grpc.reflection.v1.ServerReflection/ServerReflectionInfo",
|
||||
] {
|
||||
router
|
||||
@@ -2922,7 +2961,10 @@ mod tests {
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
assert_eq!(*calls.lock().unwrap(), vec!["openshell", "reflection"]);
|
||||
assert_eq!(
|
||||
*calls.lock().unwrap(),
|
||||
vec!["openshell", "sandbox_updates", "reflection"]
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
@@ -3018,7 +3060,10 @@ mod tests {
|
||||
.collect();
|
||||
names.sort();
|
||||
|
||||
assert_eq!(names, vec!["openshell.v1.OpenShell"]);
|
||||
assert_eq!(
|
||||
names,
|
||||
vec!["openshell.v1.OpenShell", "openshell.v1.SandboxUpdates"]
|
||||
);
|
||||
|
||||
let descriptor_response = responses.message().await.unwrap().unwrap();
|
||||
let Some(MessageResponse::FileDescriptorResponse(response)) =
|
||||
@@ -3038,6 +3083,11 @@ mod tests {
|
||||
.is_some(),
|
||||
"a fresh descriptor pool must resolve the advertised service"
|
||||
);
|
||||
assert!(
|
||||
reflected_pool
|
||||
.get_service_by_name("openshell.v1.SandboxUpdates")
|
||||
.is_some()
|
||||
);
|
||||
assert!(
|
||||
reflected_pool
|
||||
.get_message_by_name("openshell.v1.HealthRequest")
|
||||
|
||||
@@ -0,0 +1,217 @@
|
||||
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
//! Additional, fail-closed deployment checks for operator-activated image updates.
|
||||
//!
|
||||
//! These snapshots are not a durable fence or proof that old processes have
|
||||
//! stopped. Membership can expire, gateways need not advertise peer endpoints,
|
||||
//! and a backend can connect after the check. The explicit operator activation
|
||||
//! must attest that incompatible gateways have been drained and retired. The
|
||||
//! old-understood pending-operation interlock, not these observations, protects
|
||||
//! an accepted image mutation from a subsequent legacy Start.
|
||||
|
||||
use super::{ObjectRecord, PersistenceError, PersistenceResult, Store};
|
||||
use crate::gateway_members::{MEMBER_TTL, SANDBOX_UPDATE_PROTOCOL};
|
||||
use openshell_core::time::now_ms;
|
||||
|
||||
const MEMBER_OBJECT_TYPE: &str = "gateway_member";
|
||||
const MEMBER_LIMIT: u32 = 1024;
|
||||
|
||||
impl Store {
|
||||
/// Reject observable incompatible or unknown gateway deployment participants.
|
||||
///
|
||||
/// The caller separately enforces the default-off operator activation flag.
|
||||
/// `SQLite` returns success under its supported single-gateway deployment
|
||||
/// contract; this does not enforce process exclusion for a shared file.
|
||||
/// `PostgreSQL` checks fresh gateway membership records. Administrative,
|
||||
/// monitoring, and backup sessions are not gateway participants. Operator
|
||||
/// activation attests that unregistered legacy processes have been retired.
|
||||
/// Recovery of accepted operations does not use this admission check.
|
||||
pub async fn ensure_image_update_deployment_compatible(&self) -> PersistenceResult<()> {
|
||||
let Self::Postgres(_) = self else {
|
||||
return Ok(());
|
||||
};
|
||||
|
||||
// One extra row distinguishes a complete bounded snapshot from a
|
||||
// truncated listing. Never silently treat uninspected rows as compatible.
|
||||
let records = self
|
||||
.list_by_type(MEMBER_OBJECT_TYPE, MEMBER_LIMIT + 1, 0)
|
||||
.await?;
|
||||
ensure_live_members_compatible(&records, now_ms())
|
||||
}
|
||||
}
|
||||
|
||||
fn ensure_live_members_compatible(records: &[ObjectRecord], now: i64) -> PersistenceResult<()> {
|
||||
if records.len() > MEMBER_LIMIT as usize {
|
||||
return Err(PersistenceError::Config(
|
||||
"cannot establish image-update deployment compatibility: gateway membership listing exceeds its limit"
|
||||
.into(),
|
||||
));
|
||||
}
|
||||
let ttl_ms = i64::try_from(MEMBER_TTL.as_millis()).unwrap_or(i64::MAX);
|
||||
for record in records {
|
||||
// Match membership liveness semantics, including clock-skew clamping.
|
||||
// Ignoring expired records is not evidence that their processes stopped.
|
||||
let age_ms = now.saturating_sub(record.updated_at_ms).max(0);
|
||||
if age_ms >= ttl_ms {
|
||||
continue;
|
||||
}
|
||||
let member: serde_json::Map<String, serde_json::Value> =
|
||||
serde_json::from_slice(&record.payload).map_err(|_| {
|
||||
PersistenceError::Config(
|
||||
"cannot establish image-update deployment compatibility: a live gateway membership record is not a valid JSON object"
|
||||
.into(),
|
||||
)
|
||||
})?;
|
||||
let replica_id = member.get("replica_id").and_then(serde_json::Value::as_str);
|
||||
let protocol = member
|
||||
.get("sandbox_update_protocol")
|
||||
.and_then(serde_json::Value::as_u64);
|
||||
if replica_id.is_none_or(|id| id.trim().is_empty())
|
||||
|| protocol != Some(u64::from(SANDBOX_UPDATE_PROTOCOL))
|
||||
{
|
||||
return Err(PersistenceError::Config(format!(
|
||||
"sandbox updates require every live gateway membership record to advertise sandbox-update protocol {SANDBOX_UPDATE_PROTOCOL} and a nonempty replica id"
|
||||
)));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn record(payload: &[u8], updated_at_ms: i64) -> ObjectRecord {
|
||||
ObjectRecord {
|
||||
object_type: MEMBER_OBJECT_TYPE.into(),
|
||||
id: "gateway-member:gw-0".into(),
|
||||
name: "gw-0".into(),
|
||||
workspace: String::new(),
|
||||
payload: payload.to_vec(),
|
||||
created_at_ms: updated_at_ms,
|
||||
updated_at_ms,
|
||||
labels: None,
|
||||
resource_version: 1,
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn sqlite_uses_the_supported_single_gateway_contract() {
|
||||
let store = super::super::test_store().await;
|
||||
store
|
||||
.ensure_image_update_deployment_compatible()
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[ignore = "requires OPENSHELL_TEST_POSTGRES_URL; run mise run test:rust:postgres"]
|
||||
async fn postgres_activation_ignores_administrative_sessions_and_preserves_application_name() {
|
||||
use super::super::test_postgres::TestSchema;
|
||||
let schema = TestSchema::create("sandbox_update_activation").await;
|
||||
let mut url = url::Url::parse(schema.url()).unwrap();
|
||||
url.query_pairs_mut()
|
||||
.append_pair("application_name", "operator-custom-name");
|
||||
let store = Store::connect(url.as_str()).await.unwrap();
|
||||
let Store::Postgres(pg) = &store else {
|
||||
unreachable!()
|
||||
};
|
||||
let name: String = sqlx::query_scalar("SHOW application_name")
|
||||
.fetch_one(&pg.pool)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(name, "operator-custom-name");
|
||||
// TestSchema's unmarked admin connection remains open in this database.
|
||||
store
|
||||
.ensure_image_update_deployment_compatible()
|
||||
.await
|
||||
.unwrap();
|
||||
let index = crate::gateway_members::GatewayMemberIndex::new(
|
||||
std::sync::Arc::new(store.clone()),
|
||||
MEMBER_TTL,
|
||||
);
|
||||
index
|
||||
.register("compatible", "https://gateway:8443")
|
||||
.await
|
||||
.unwrap();
|
||||
store
|
||||
.ensure_image_update_deployment_compatible()
|
||||
.await
|
||||
.unwrap();
|
||||
store
|
||||
.put_if(
|
||||
MEMBER_OBJECT_TYPE,
|
||||
"gateway-member:legacy",
|
||||
"legacy",
|
||||
"",
|
||||
br#"{"replica_id":"legacy"}"#,
|
||||
None,
|
||||
super::super::WriteCondition::MustCreate,
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(matches!(
|
||||
store.ensure_image_update_deployment_compatible().await,
|
||||
Err(PersistenceError::Config(_))
|
||||
));
|
||||
store.close().await;
|
||||
schema.drop_schema().await;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compatible_live_members_and_empty_snapshot_pass() {
|
||||
assert!(ensure_live_members_compatible(&[], 100).is_ok());
|
||||
let member = record(
|
||||
br#"{"replica_id":"gw-0","peer_endpoint":"https://gw-0:8443","sandbox_update_protocol":1}"#,
|
||||
100,
|
||||
);
|
||||
assert!(ensure_live_members_compatible(&[member], 100).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn malformed_missing_unknown_or_ill_typed_protocols_fail_closed() {
|
||||
let invalid: &[&[u8]] = &[
|
||||
b"not JSON",
|
||||
br"[]",
|
||||
br#"["gw-0",1]"#,
|
||||
br"null",
|
||||
br#"{"replica_id":"gw-0"}"#,
|
||||
br#"{"replica_id":"gw-0","sandbox_update_protocol":0}"#,
|
||||
br#"{"replica_id":"gw-0","sandbox_update_protocol":2}"#,
|
||||
br#"{"replica_id":"gw-0","sandbox_update_protocol":"1"}"#,
|
||||
br#"{"replica_id":"gw-0","sandbox_update_protocol":true}"#,
|
||||
br#"{"replica_id":"gw-0","sandbox_update_protocol":1.0}"#,
|
||||
br#"{"sandbox_update_protocol":1}"#,
|
||||
br#"{"replica_id":"","sandbox_update_protocol":1}"#,
|
||||
br#"{"replica_id":" ","sandbox_update_protocol":1}"#,
|
||||
];
|
||||
for payload in invalid {
|
||||
assert!(ensure_live_members_compatible(&[record(payload, 100)], 100).is_err());
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn expired_records_are_not_considered_live_but_future_timestamps_are() {
|
||||
let ttl_ms = i64::try_from(MEMBER_TTL.as_millis()).unwrap();
|
||||
assert!(ensure_live_members_compatible(&[record(b"legacy", 100)], 100 + ttl_ms).is_ok());
|
||||
assert!(
|
||||
ensure_live_members_compatible(&[record(b"legacy", 100)], 100 + ttl_ms - 1).is_err()
|
||||
);
|
||||
assert!(ensure_live_members_compatible(&[record(b"legacy", 200)], 100).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn any_incompatible_live_member_rejects_the_snapshot() {
|
||||
let compatible = record(br#"{"replica_id":"gw-0","sandbox_update_protocol":1}"#, 100);
|
||||
let legacy = record(br#"{"replica_id":"gw-1"}"#, 100);
|
||||
assert!(ensure_live_members_compatible(&[compatible, legacy], 100).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn listing_truncation_fails_closed_even_when_extra_rows_are_expired() {
|
||||
let records = vec![record(b"legacy", 0); MEMBER_LIMIT as usize + 1];
|
||||
let now = i64::try_from(MEMBER_TTL.as_millis()).unwrap();
|
||||
assert!(ensure_live_members_compatible(&records, now).is_err());
|
||||
}
|
||||
}
|
||||
@@ -3,12 +3,14 @@
|
||||
|
||||
//! Persistence layer for `OpenShell` Server.
|
||||
|
||||
mod image_activation;
|
||||
mod legacy_time_wire;
|
||||
pub mod lock_order;
|
||||
pub mod mutation_lock;
|
||||
mod postgres;
|
||||
#[cfg(test)]
|
||||
mod refresh_lock_tests;
|
||||
mod sandbox_image;
|
||||
mod sqlite;
|
||||
|
||||
pub use crate::storage_proto::{
|
||||
|
||||
@@ -40,7 +40,7 @@ use super::{DELETE_MANY_BATCH_SIZE, DRAFT_CHUNK_OBJECT_TYPE, POLICY_OBJECT_TYPE}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct PostgresStore {
|
||||
pool: PgPool,
|
||||
pub(super) pool: PgPool,
|
||||
/// Dedicated connections for session-level mutation advisory locks.
|
||||
lock_pool: PgPool,
|
||||
/// How acquisitions and guards use the lock pool's connections.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -48,7 +48,7 @@ pub const DEFAULT_MAX_CONNECTIONS: u32 = 5;
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct SqliteStore {
|
||||
pool: SqlitePool,
|
||||
pub(super) pool: SqlitePool,
|
||||
/// Pool for writes whose loss after a crash is harmless; see
|
||||
/// [`SqliteStore::create_relaxed`]. On-disk stores open it with
|
||||
/// `synchronous=NORMAL`; in-memory stores share `pool`.
|
||||
|
||||
@@ -28,7 +28,7 @@ use openshell_core::Config;
|
||||
use crate::multiplex::GrpcRateLimiter;
|
||||
|
||||
const REFLECTED_PROTO_ROOTS: &[&str] = &["openshell.proto"];
|
||||
const ADVERTISED_SERVICES: &[&str] = &["openshell.v1.OpenShell"];
|
||||
const ADVERTISED_SERVICES: &[&str] = &["openshell.v1.OpenShell", "openshell.v1.SandboxUpdates"];
|
||||
|
||||
pub type GatewayReflectionServer = ServerReflectionServer<GatewayReflectionService>;
|
||||
|
||||
|
||||
@@ -15,6 +15,11 @@
|
||||
|
||||
include!(concat!(env!("OUT_DIR"), "/openshell.storage.v1.rs"));
|
||||
|
||||
mod sandbox_image {
|
||||
include!(concat!(env!("OUT_DIR"), "/openshell.storage.image.v1.rs"));
|
||||
}
|
||||
pub(crate) use sandbox_image::{StoredSandboxImageState, StoredSandboxImageUpdate};
|
||||
|
||||
pub(crate) const STORAGE_FILE_DESCRIPTOR_SET: &[u8] =
|
||||
include_bytes!(concat!(env!("OUT_DIR"), "/storage_descriptor.bin"));
|
||||
|
||||
@@ -163,12 +168,21 @@ mod tests {
|
||||
// and SUSPENDED values to the shared
|
||||
// SandboxPhase enum. Stored rows never carried those values, so prior
|
||||
// payloads decode unchanged.
|
||||
|
||||
// Image updates add a separate public service/request and a new private
|
||||
// image-state format. Existing public status and frozen storage.v1 stay
|
||||
// unchanged; only the private durable closure grows.
|
||||
// Configuration updates add presence-aware argv/map wrappers shared with
|
||||
// private durable intent. Existing image-only payloads decode with all four
|
||||
// selections absent; storage.v1 remains frozen. Public field 3 retains its
|
||||
// image wire identity, now with explicit presence. Compute configuration
|
||||
// updates use a separate capability/service and add one classified RPC.
|
||||
const PUBLIC_RPC_SCHEMA_SHA256: &str =
|
||||
"f24c9b77a85f59ae9ab60f783d45654d78bf566fcf260f9131f1aba39fa4b59b";
|
||||
"be99bb3fcc7894eb85d774f54b642173e70dd37bc8e5fcaf3fdff2454ae689e1";
|
||||
const DURABLE_SCHEMA_SHA256: &str =
|
||||
"ffb20595e36032a9476d1c6956e2dcdc814f85286fc4735751e5ec6f499ea9ed";
|
||||
"cb15f8f79a855064a43fe8c6cd908bf07eca6604e2bf8ab7d2de0ef33e97da4e";
|
||||
const PUBLIC_DURABLE_OVERLAP_SHA256: &str =
|
||||
"ac1ffc4b5b81467de27270a4e33f6db1e31eeb2c52f79750dbaaf19515eadbb2";
|
||||
"c0c3ea17f105aaa6d843c466fd4240451c2105dce7675384935d4aaecdac8abf";
|
||||
// Encoded with the schema that omitted stable_placeholder. Keep these
|
||||
// bytes fixed so the current encoder cannot hide a compatibility change.
|
||||
const LEGACY_STATIC_PROFILE: &str = "0a240a116c65676163792d70726f66696c652d6964120d6c65676163792d737461746963280712510a0d6c65676163792d73746174696312184c6567616379207374617469632063726564656e7469616c2a260a076170695f6b65791a1153594e5448455449435f4150495f4b455920012a06626561726572";
|
||||
@@ -229,6 +243,7 @@ mod tests {
|
||||
".openshell.storage.v1.StoredConfigUpdateOperation",
|
||||
".openshell.storage.v1.StoredProviderCredentialRefreshStateV2",
|
||||
".openshell.storage.v1.StoredProviderProfile",
|
||||
".openshell.storage.image.v1.StoredSandboxImageState",
|
||||
".openshell.v1.Sandbox",
|
||||
".openshell.v1.SandboxWorkloadTemplate",
|
||||
".openshell.v1.ServiceEndpoint",
|
||||
@@ -471,7 +486,12 @@ mod tests {
|
||||
let public = FileDescriptorSet::decode(openshell_core::FILE_DESCRIPTOR_SET)
|
||||
.expect("public descriptor set must decode");
|
||||
for file in public.file {
|
||||
assert_ne!(file.package.as_deref(), Some("openshell.storage.v1"));
|
||||
assert!(
|
||||
!file
|
||||
.package
|
||||
.as_deref()
|
||||
.is_some_and(|package| package.starts_with("openshell.storage."))
|
||||
);
|
||||
for message in file.message_type {
|
||||
let name = message.name.expect("message name");
|
||||
assert!(
|
||||
@@ -576,7 +596,8 @@ mod tests {
|
||||
}
|
||||
if !matches!(
|
||||
(package, service_name),
|
||||
("openshell.v1", "OpenShell") | ("openshell.inference.v1", "Inference")
|
||||
("openshell.v1", "OpenShell" | "SandboxUpdates")
|
||||
| ("openshell.inference.v1", "Inference")
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
@@ -631,7 +652,7 @@ mod tests {
|
||||
}
|
||||
assert_eq!(
|
||||
compiled_method_count,
|
||||
106 + PROVIDER_READINESS_RPC_SIGNATURES.len()
|
||||
108 + PROVIDER_READINESS_RPC_SIGNATURES.len()
|
||||
+ PEER_OWNER_RPC_SIGNATURES.len()
|
||||
+ PEER_CONFIG_RPC_SIGNATURES.len()
|
||||
+ isolation_methods.len()
|
||||
@@ -641,7 +662,7 @@ mod tests {
|
||||
);
|
||||
assert_eq!(
|
||||
methods.len(),
|
||||
80 + PROVIDER_READINESS_RPC_SIGNATURES.len()
|
||||
81 + PROVIDER_READINESS_RPC_SIGNATURES.len()
|
||||
+ PEER_OWNER_RPC_SIGNATURES.len()
|
||||
+ PEER_CONFIG_RPC_SIGNATURES.len(),
|
||||
"inventory every public gateway RPC"
|
||||
@@ -702,9 +723,9 @@ mod tests {
|
||||
overlap_hash.as_str(),
|
||||
),
|
||||
(
|
||||
(330, 31),
|
||||
(95, 22),
|
||||
(83, 22),
|
||||
(334, 31),
|
||||
(100, 22),
|
||||
(86, 22),
|
||||
PUBLIC_RPC_SCHEMA_SHA256,
|
||||
DURABLE_SCHEMA_SHA256,
|
||||
PUBLIC_DURABLE_OVERLAP_SHA256
|
||||
|
||||
@@ -4810,7 +4810,11 @@ mod tests {
|
||||
let launching = fixture.replica("gw-b");
|
||||
let (pass, launched) = tokio::join!(
|
||||
pool.reconcile_once(),
|
||||
launching.place_launch(launch("rt-route-new0", "sandbox-new", "generation-1")),
|
||||
Box::pin(launching.place_launch(launch(
|
||||
"rt-route-new0",
|
||||
"sandbox-new",
|
||||
"generation-1"
|
||||
))),
|
||||
);
|
||||
pass.unwrap();
|
||||
// Every route is re-adopted by the new process at the next term,
|
||||
|
||||
@@ -105,6 +105,12 @@ pub enum FakeComputeDriverCall {
|
||||
CreateSandbox {
|
||||
sandbox: Option<DriverSandbox>,
|
||||
},
|
||||
UpdateSandboxConfiguration {
|
||||
request: openshell_core::proto::compute::v1::UpdateSandboxConfigurationRequest,
|
||||
},
|
||||
UpdateSandboxImage {
|
||||
request: openshell_core::proto::compute::v1::SandboxUpdatePreconditions,
|
||||
},
|
||||
StopSandbox {
|
||||
sandbox_id: String,
|
||||
sandbox_name: String,
|
||||
@@ -140,6 +146,9 @@ struct FakeComputeDriverState {
|
||||
sandboxes: HashMap<String, DriverSandbox>,
|
||||
calls: Vec<FakeComputeDriverCall>,
|
||||
traceparents: Vec<String>,
|
||||
image_update_error: Option<tonic::Code>,
|
||||
image_validation_error: Option<tonic::Code>,
|
||||
image_update_pause: Option<(Arc<tokio::sync::Notify>, Arc<tokio::sync::Semaphore>)>,
|
||||
}
|
||||
|
||||
impl Default for FakeComputeDriver {
|
||||
@@ -183,6 +192,9 @@ impl FakeComputeDriver {
|
||||
sandboxes: HashMap::new(),
|
||||
calls: Vec::new(),
|
||||
traceparents: Vec::new(),
|
||||
image_update_error: None,
|
||||
image_validation_error: None,
|
||||
image_update_pause: None,
|
||||
})),
|
||||
}
|
||||
}
|
||||
@@ -225,6 +237,22 @@ impl FakeComputeDriver {
|
||||
self
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn with_sandbox_image_updates(self) -> Self {
|
||||
self.with_state(|state| {
|
||||
state
|
||||
.capabilities
|
||||
.extension
|
||||
.as_mut()
|
||||
.unwrap()
|
||||
.supported_capabilities
|
||||
.push(
|
||||
openshell_core::extension_protocol::COMPUTE_UPDATE_SANDBOX_CONFIGURATION.into(),
|
||||
);
|
||||
});
|
||||
self
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn calls(&self) -> Vec<FakeComputeDriverCall> {
|
||||
self.with_state(|state| state.calls.clone())
|
||||
@@ -253,13 +281,31 @@ impl FakeComputeDriver {
|
||||
let driver = self.clone();
|
||||
let task = tokio::spawn(async move {
|
||||
tonic::transport::Server::builder()
|
||||
.add_service(ComputeDriverServer::new(driver))
|
||||
.add_service(ComputeDriverServer::new(driver.clone()))
|
||||
.add_service(openshell_core::proto::compute::v1::compute_driver_configuration_updates_server::ComputeDriverConfigurationUpdatesServer::new(driver.clone()))
|
||||
.serve_with_incoming(UnixIncoming { listener })
|
||||
.await
|
||||
});
|
||||
Ok(FakeComputeDriverServerHandle { socket_path, task })
|
||||
}
|
||||
|
||||
pub fn pause_image_updates(&self) -> (Arc<tokio::sync::Notify>, Arc<tokio::sync::Semaphore>) {
|
||||
let entered = Arc::new(tokio::sync::Notify::new());
|
||||
let release = Arc::new(tokio::sync::Semaphore::new(0));
|
||||
self.with_state(|state| {
|
||||
state.image_update_pause = Some((entered.clone(), release.clone()));
|
||||
});
|
||||
(entered, release)
|
||||
}
|
||||
|
||||
pub fn set_image_update_error(&self, code: Option<tonic::Code>) {
|
||||
self.with_state(|state| state.image_update_error = code);
|
||||
}
|
||||
|
||||
pub fn set_image_validation_error(&self, code: Option<tonic::Code>) {
|
||||
self.with_state(|state| state.image_validation_error = code);
|
||||
}
|
||||
|
||||
fn with_state<R>(&self, f: impl FnOnce(&mut FakeComputeDriverState) -> R) -> R {
|
||||
let mut state = self
|
||||
.state
|
||||
@@ -310,6 +356,50 @@ impl Stream for UnixIncoming {
|
||||
}
|
||||
}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl openshell_core::compute::ComputeDriver for FakeComputeDriver {
|
||||
async fn update_sandbox_configuration(
|
||||
&self,
|
||||
request: Request<openshell_core::proto::compute::v1::UpdateSandboxConfigurationRequest>,
|
||||
) -> Result<
|
||||
Response<openshell_core::proto::compute::v1::UpdateSandboxConfigurationResponse>,
|
||||
Status,
|
||||
> {
|
||||
let pause = self.with_state(|state| state.image_update_pause.clone());
|
||||
if let Some((entered, release)) = pause {
|
||||
entered.notify_one();
|
||||
release.acquire().await.unwrap().forget();
|
||||
}
|
||||
self.record_traceparent(request.metadata());
|
||||
self.with_state(|state| {
|
||||
let request = request.into_inner();
|
||||
let validate_only = request.preconditions.as_ref().unwrap().validate_only;
|
||||
state.calls.push(FakeComputeDriverCall::UpdateSandboxImage {
|
||||
request: request.preconditions.clone().unwrap(),
|
||||
});
|
||||
state
|
||||
.calls
|
||||
.push(FakeComputeDriverCall::UpdateSandboxConfiguration { request });
|
||||
if let Some(code) = state.image_validation_error.filter(|_| validate_only) {
|
||||
return Err(Status::new(code, "injected configuration validation error"));
|
||||
}
|
||||
if let Some(code) = state.image_update_error.filter(|_| !validate_only) {
|
||||
return Err(Status::new(code, "injected configuration update error"));
|
||||
}
|
||||
Ok(Response::new(
|
||||
openshell_core::proto::compute::v1::UpdateSandboxConfigurationResponse {},
|
||||
))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl openshell_core::proto::compute::v1::compute_driver_configuration_updates_server::ComputeDriverConfigurationUpdates for FakeComputeDriver {
|
||||
async fn update_sandbox_configuration(&self, request: Request<openshell_core::proto::compute::v1::UpdateSandboxConfigurationRequest>) -> Result<Response<openshell_core::proto::compute::v1::UpdateSandboxConfigurationResponse>, Status> {
|
||||
openshell_core::compute::ComputeDriver::update_sandbox_configuration(self, request).await
|
||||
}
|
||||
}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl ComputeDriver for FakeComputeDriver {
|
||||
async fn authenticate_sandbox(
|
||||
|
||||
@@ -42,7 +42,20 @@ rules:
|
||||
resources:
|
||||
- pods
|
||||
verbs:
|
||||
- create
|
||||
- delete
|
||||
- get
|
||||
# Updates must rule out every remaining workload Pod, including Pods
|
||||
# identified only by owner UID or the controller's recorded name.
|
||||
- list
|
||||
- patch
|
||||
- watch
|
||||
- apiGroups: [""]
|
||||
resources: ["services"]
|
||||
verbs: ["create", "get"]
|
||||
- apiGroups: ["networking.k8s.io"]
|
||||
resources: ["networkpolicies"]
|
||||
verbs: ["create", "get"]
|
||||
# Generation-scoped sandbox bootstrap Secrets.
|
||||
- apiGroups:
|
||||
- ""
|
||||
|
||||
@@ -12,6 +12,28 @@ release:
|
||||
namespace: app-a
|
||||
|
||||
tests:
|
||||
- it: grants supervisor lifecycle and complete stopped-update pod inspection
|
||||
template: templates/role.yaml
|
||||
asserts:
|
||||
- contains:
|
||||
path: rules
|
||||
content:
|
||||
apiGroups: [""]
|
||||
resources: ["pods"]
|
||||
verbs: ["create", "delete", "get", "list", "patch", "watch"]
|
||||
- contains:
|
||||
path: rules
|
||||
content:
|
||||
apiGroups: [""]
|
||||
resources: ["services"]
|
||||
verbs: ["create", "get"]
|
||||
- contains:
|
||||
path: rules
|
||||
content:
|
||||
apiGroups: ["networking.k8s.io"]
|
||||
resources: ["networkpolicies"]
|
||||
verbs: ["create", "get"]
|
||||
|
||||
- it: creates the workspace role in the release namespace
|
||||
template: templates/role.yaml
|
||||
asserts:
|
||||
|
||||
@@ -255,6 +255,9 @@ metrics_bind_address = "0.0.0.0:9090"
|
||||
|
||||
log_level = "info"
|
||||
|
||||
# Activate only after retiring incompatible gateways and outstanding driver work.
|
||||
enable_sandbox_updates = false
|
||||
|
||||
# Connection ceiling for the gateway's database pool. Omit to use the backend
|
||||
# default: 10 for Postgres, 5 for on-disk SQLite.
|
||||
database_max_connections = 10
|
||||
@@ -499,6 +502,46 @@ Changing the installation name, the hostname format, or both changes every
|
||||
hash URL issued before the change; previously issued URLs stop resolving
|
||||
until callers request a fresh URL.
|
||||
|
||||
## Sandbox Updates
|
||||
|
||||
`[openshell.gateway] enable_sandbox_updates` defaults to `false`. It gates
|
||||
the separate `openshell.v1.SandboxUpdates/UpdateSandbox` RPC on the existing
|
||||
gateway listener. It gates image, command, environment, label, and annotation
|
||||
updates for stopped sandboxes. Upgrade every gateway before activation.
|
||||
Disabling the flag stops new updates; accepted updates still recover automatically.
|
||||
The equivalent gateway process flag and environment variable
|
||||
are `--enable-sandbox-updates true` and
|
||||
`OPENSHELL_ENABLE_SANDBOX_UPDATES=true`.
|
||||
|
||||
Enable it only after completing the
|
||||
[activation checklist](/kubernetes/high-availability#activate-sandbox-updates).
|
||||
Setting it to `true` attests that every gateway that can write the database is
|
||||
compatible and incompatible gateways cannot return. Keep it false throughout a
|
||||
rolling upgrade, including while outstanding operations from retired gateways
|
||||
settle. Gateway membership checks fail closed but do not prove retirement. PostgreSQL
|
||||
administrative and monitoring sessions do not block updates; connection
|
||||
`application_name` settings are preserved.
|
||||
|
||||
For a schema-v2 file, set:
|
||||
|
||||
```toml
|
||||
[openshell.gateway]
|
||||
enable_sandbox_updates = true
|
||||
```
|
||||
|
||||
The canonical Helm values form uses a dotted table key:
|
||||
|
||||
```yaml
|
||||
gatewayConfig:
|
||||
"openshell.gateway":
|
||||
enable_sandbox_updates: true
|
||||
```
|
||||
|
||||
This is not an automatic mixed-version migration switch. Do not downgrade while
|
||||
an image update is pending or retained sandboxes require the private image
|
||||
revision protocol. SQLite supports one gateway process only; activation attests
|
||||
that the previous process has stopped, not that sharing a SQLite file enables HA.
|
||||
|
||||
## Database Pool
|
||||
|
||||
`[openshell.gateway] database_max_connections` sets the connection ceiling for the gateway's database pool. One pool is shared by every database-backed RPC, so this value is also the gateway's ceiling on concurrent database work. When the pool is saturated, callers queue on connection acquisition and sqlx logs that the time to acquire a connection exceeded its slow threshold. The background readiness monitor draws from the same pool, so sustained saturation can push its database ping past the check timeout and flip `/readyz` to not-ready while the gateway is otherwise serving.
|
||||
|
||||
@@ -1006,10 +1006,130 @@ command. Starting a fresh instance invalidates SSH sessions issued for the
|
||||
previous runtime generation. Delete an inactive sandbox normally when you no
|
||||
longer need its retained state.
|
||||
|
||||
To change the configuration of a retained `Completed` or `Error/MainProcessFailed`
|
||||
sandbox, run `sandbox stop` first. Stop cleans up compute and moves the sandbox
|
||||
to `Stopped`, preserving its exit code until the next start. This stop behavior
|
||||
also applies when sandbox updates are disabled.
|
||||
|
||||
If the terminal sandbox's compute resource is already gone, stop succeeds and
|
||||
preserves its terminal phase and exit code. Configuration updates require a
|
||||
retained compute resource.
|
||||
|
||||
If an infrastructure failure leaves a sandbox in `Error`, fix the reported cause, then run `sandbox stop` followed by `sandbox start`. Wait for stop to succeed before starting. This retains the sandbox and its persistent workspace on backends that support restarting stopped sandboxes; it cannot recover deleted resources or storage.
|
||||
|
||||
For a provisioning timeout, wait for cleanup to finish, then run `sandbox start` directly.
|
||||
|
||||
### Update the Image of a Stopped Sandbox
|
||||
|
||||
Use `openshell.v1.SandboxUpdates/UpdateSandbox` to change a stopped sandbox's
|
||||
workload image, command, environment overrides, labels, and annotations together. This separate service uses the same gateway listener
|
||||
and authentication as `OpenShell`; it requires `sandbox:write` and workspace
|
||||
user access. This operation initially supports the Kubernetes compute driver
|
||||
for workload changes; metadata-only updates work with every compute driver. There is no
|
||||
`sandbox update` CLI command or curated SDK update method.
|
||||
|
||||
Image updates default to disabled. Ask the operator to complete the
|
||||
[compatible deployment rollout](/kubernetes/high-availability#activate-sandbox-updates)
|
||||
before enabling `enable_sandbox_updates` in the
|
||||
[gateway configuration](/how-it-works/gateways/configuration#sandbox-updates).
|
||||
|
||||
The update preserves the sandbox ID, Kubernetes custom resource, and persistent
|
||||
volume claim. It does not delete and recreate the sandbox, reseed its existing
|
||||
workspace from the new image, or change the trusted supervisor and sandbox
|
||||
runtime images. Omitted fields retain their current values. A supplied command
|
||||
replaces the argv list without shell expansion; an empty list restores the default
|
||||
login shell resolved from the workload image. The Kubernetes driver rejects an
|
||||
update before acceptance if the encoded command and other custom-resource
|
||||
annotations exceed Kubernetes' combined 256 KiB annotation limit.
|
||||
A supplied environment map replaces
|
||||
sandbox-level overrides; clearing it restores any template-provided defaults.
|
||||
Labels and annotations replace user metadata on the gateway resource, preserve
|
||||
system-owned keys, and do not set Kubernetes pod metadata. Attempts to supply
|
||||
system-owned keys are rejected. Metadata-only changes commit directly without
|
||||
contacting the compute driver. The retention annotation is preserved when omitted.
|
||||
Annotation changes accepted through `UpdateConfig` after the sandbox update is
|
||||
accepted remain in place when the update completes, including after a retry.
|
||||
|
||||
Without a pending update, a request whose target values already match is a no-op.
|
||||
Use a different image tag or an immutable digest to select new image content.
|
||||
Workload changes require `openshell.compute.update-sandbox-configuration`.
|
||||
Upgrade every gateway before enabling updates.
|
||||
|
||||
Follow this sequence through the CLI and a generated/raw RPC client.
|
||||
|
||||
1. Run `openshell sandbox stop my-sandbox` and wait for `Stopped`.
|
||||
2. Call `GetSandbox` with the named `workspace_scope` and sandbox `name`. Read
|
||||
`sandbox.metadata.resource_version` after stop completes.
|
||||
3. Call `UpdateSandbox` with the same scope and name, the selected update fields,
|
||||
and `expected_resource_version` set to that current, nonzero
|
||||
version. The response is a `SandboxResponse`; the sandbox remains stopped.
|
||||
4. After the update succeeds, run `openshell sandbox start my-sandbox` to launch
|
||||
the new image.
|
||||
|
||||
With an authenticated Rust SDK `OpenShellClient`, use the separate raw client
|
||||
and the `current_resource_version` read in step 2:
|
||||
|
||||
```rust
|
||||
use openshell_sdk::raw::{UpdateSandboxRequest, proto};
|
||||
|
||||
let mut updates = client.raw_sandbox_updates_fresh().await?;
|
||||
let response = updates.update_sandbox(UpdateSandboxRequest {
|
||||
workspace_scope: Some(proto::workspace_selector("default")),
|
||||
name: "my-sandbox".into(),
|
||||
image: Some("registry.example.com/agents/my-agent:2.0".into()),
|
||||
command: Some(proto::SandboxCommandUpdate {
|
||||
argv: vec!["/usr/local/bin/start-agent".into(), "--new-option".into()],
|
||||
}),
|
||||
environment: Some(proto::SandboxMapUpdate {
|
||||
values: [("AGENT_MODE".into(), "updated".into())].into_iter().collect(),
|
||||
}),
|
||||
expected_resource_version: current_resource_version,
|
||||
request_id: String::new(),
|
||||
..Default::default()
|
||||
}).await?;
|
||||
```
|
||||
|
||||
The helper shares the main client's transport and live authentication slot.
|
||||
Generated clients use the `SandboxUpdates` service, not `OpenShell`.
|
||||
|
||||
If the new image fails to start and provisioning times out, wait until
|
||||
`provisioning.cleanup_completed_time` is present and
|
||||
`provisioning.driver_operation_pending` is false. Run `sandbox stop` to move the
|
||||
reclaimed sandbox from `Error` to `Stopped`, then repeat the update sequence with
|
||||
a working image. Stop retains the timeout record and storage; it does not stop
|
||||
compute again. This Stop behavior also applies when updates are disabled.
|
||||
Cleanup that is still pending cannot be bypassed.
|
||||
|
||||
An `ABORTED` version conflict requires another read before you construct a new
|
||||
conditional request. The gateway limits driver preflight and each accepted update
|
||||
attempt to one minute each. A timed-out preflight does not accept an update.
|
||||
An accepted update continues after client cancellation until it completes or
|
||||
times out. A timeout releases the sandbox for recovery or deletion; it does not
|
||||
roll back an update the driver may have applied.
|
||||
An update failure or disconnected response can leave a
|
||||
pending operation, even when `sandbox.spec.template.image` already names the
|
||||
target image. Start, stop, and provisioning retry remain blocked until reconciliation
|
||||
succeeds. The gateway retries accepted updates at startup and periodically, even
|
||||
if new updates are disabled. Failed automatic attempts wait 1, 2, 4, 8, then
|
||||
15 minutes before another attempt becomes eligible. Recovery continues at the
|
||||
15-minute limit; periodic scans can add up to a minute to the wait. Each gateway
|
||||
retries independently, and restarting a gateway resets its retry delay.
|
||||
If the driver is unavailable or no longer supports
|
||||
updates, restore a compatible driver. Explicit deletion can reclaim a sandbox
|
||||
whose update cannot complete; deletion also removes its stored workspace.
|
||||
|
||||
To resume immediately, read the current resource version and call `UpdateSandbox`
|
||||
with the same scope and name, that version, and no selected fields. The gateway
|
||||
uses the stored operation and target values. With no pending update, this returns
|
||||
`FAILED_PRECONDITION`. The operation ID is visible in
|
||||
`status.provisioning.driver_operation_id` while pending; the full intent stays private.
|
||||
|
||||
The optional `request_id` uses [durable request admission](/sdk/api-errors#durable-request-admission).
|
||||
Reusing an unresolved ID returns `REQUEST_OUTCOME_UNCERTAIN`. For explicit resume,
|
||||
use a new ID or omit it, and use the fresh resource version. Do not change the
|
||||
original payload under its original ID. A confirmed successful request still
|
||||
replays without applying another update.
|
||||
|
||||
## Delete Sandboxes
|
||||
|
||||
Deleting a sandbox stops all processes, releases resources, and purges injected credentials.
|
||||
|
||||
@@ -504,6 +504,45 @@ during the upgrade, the older replicas serialize every mutation across the
|
||||
fleet. Mutations can be slower and can briefly fail under load until the
|
||||
rollout finishes.
|
||||
|
||||
### Activate Sandbox Updates
|
||||
|
||||
Sandbox configuration updates default to disabled. Setting
|
||||
`[openshell.gateway] enable_sandbox_updates = true` is an operator
|
||||
attestation that the deployment uniformly supports the image-update protocol.
|
||||
Use this sequence before activating the
|
||||
[separate update service](/how-it-works/sandboxes/overview#update-the-image-of-a-stopped-sandbox).
|
||||
|
||||
1. Keep the setting false throughout the rolling upgrade. Drain and retire
|
||||
every incompatible gateway, including unregistered processes and Jobs, and
|
||||
settle all outstanding driver operations they started. A completed
|
||||
Kubernetes rollout alone does not establish this.
|
||||
2. Prevent incompatible gateways from returning. Pin compatible images and
|
||||
retire old rollback targets, Jobs, and other launch configurations that could
|
||||
reconnect to the database.
|
||||
3. Confirm live gateway membership records advertise sandbox-update protocol 1.
|
||||
Missing or incompatible capabilities, malformed records, and truncated
|
||||
membership reads reject updates. These observations are not durable proof
|
||||
of fleet compatibility. Expiry or absence of a membership record does not
|
||||
prove that its process and driver work have stopped.
|
||||
4. Activate the flag only after completing retirement and preventing old
|
||||
gateways from returning. See the exact
|
||||
[TOML, environment, process flag, and Helm forms](/how-it-works/gateways/configuration#sandbox-updates).
|
||||
|
||||
A pending image update atomically sets the existing
|
||||
`provisioning.driver_operation_pending` and `provisioning.driver_operation_id`
|
||||
fields alongside private image intent. Start, stop, and provisioning-retry paths honor this per-sandbox barrier, including after a
|
||||
legacy public record round-trip. Successful reconciliation preserves current
|
||||
provisioning diagnostics and clocks, clears the pending flag, and retains the
|
||||
new operation ID to fence delayed callbacks. Previously absent provisioning
|
||||
remains absent unless concurrent writes added diagnostics or clocks. This
|
||||
barrier does not replace the deployment activation checklist.
|
||||
Image intent and revision are not new public `SandboxStatus` fields.
|
||||
|
||||
Do not downgrade while updates are pending or retained sandboxes require the
|
||||
private revision protocol. SQLite supports one gateway process only. For a
|
||||
SQLite gateway, activation attests that the old process has stopped; a shared
|
||||
SQLite file is not an HA deployment.
|
||||
|
||||
## (Experimental) Multi-tenant supervisor
|
||||
|
||||
By default, the Kubernetes driver runs a supervisor Pod for each sandbox. With
|
||||
|
||||
@@ -409,13 +409,19 @@ managed mode, the configured image-pull Secrets. The gateway stages their
|
||||
contents into each sandbox runtime generation's Secrets in the workspace
|
||||
namespace.
|
||||
|
||||
The namespaced Role covers sandbox lifecycle and identity:
|
||||
The namespaced Role covers sandbox lifecycle, supervisor resources, and identity.
|
||||
Both the gateway chart and separate workspace chart grant these permissions.
|
||||
Stopped-sandbox updates use direct Pod `get` requests to check known workload and
|
||||
supervisor Pod names for remaining Pods.
|
||||
|
||||
| API Group | Resource | Verbs |
|
||||
|---|---|---|
|
||||
| `agents.x-k8s.io` | `sandboxes`, `sandboxes/status` | create, delete, get, list, patch, update, watch |
|
||||
| `""` | `events` | get, list, watch |
|
||||
| `""` | `pods` | get |
|
||||
| `""` | `pods` | create, delete, get, list, patch, watch |
|
||||
| `""` | `services` | create, get |
|
||||
| `""` | `secrets` | create, delete |
|
||||
| `networking.k8s.io` | `networkpolicies` | create, get |
|
||||
|
||||
The ClusterRole grants node inspection and token validation:
|
||||
|
||||
|
||||
+36
-3
@@ -56,7 +56,7 @@ guarantee.
|
||||
| `ABORTED` | Read fresh state before retrying a conditional operation. |
|
||||
| `UNAVAILABLE`, `RESOURCE_EXHAUSTED` | A transient condition may recover. Apply backoff and any minimum retry delay only to a retry-safe operation. |
|
||||
| `DEADLINE_EXCEEDED`, `CANCELLED` | A mutation's outcome can be unknown. Cancellation does not imply rollback. |
|
||||
| `UNIMPLEMENTED` | Check gateway and SDK version compatibility. |
|
||||
| `UNIMPLEMENTED` | Check gateway and SDK version compatibility and whether the compute driver supports the operation. |
|
||||
| `INTERNAL`, `UNKNOWN`, `DATA_LOSS` | Preserve the status and correlation metadata for diagnosis. Do not blindly retry mutations. |
|
||||
|
||||
## SDK access
|
||||
@@ -92,19 +92,23 @@ No SDK automatically retries a mutation as a result of decoding these details.
|
||||
|
||||
## Durable request admission
|
||||
|
||||
Thirty user-callable unary RPCs accept an optional `request_id` in their protobuf requests.
|
||||
Thirty-one user-callable unary RPCs accept an optional `request_id` in their protobuf requests.
|
||||
|
||||
| Resource | RPCs |
|
||||
|---|---|
|
||||
| Workspace | `CreateWorkspace`, `DeleteWorkspace` |
|
||||
| Workspace member | `AddWorkspaceMember`, `RemoveWorkspaceMember` |
|
||||
| Sandbox template | `CreateSandboxTemplate`, `DeleteSandboxTemplate` |
|
||||
| Sandbox | `CreateSandbox`, `DeleteSandbox`, `StartSandbox`, `StopSandbox`, `AttachSandboxProvider`, `DetachSandboxProvider` |
|
||||
| Sandbox | `CreateSandbox`, `UpdateSandbox`, `DeleteSandbox`, `StartSandbox`, `StopSandbox`, `AttachSandboxProvider`, `DetachSandboxProvider` |
|
||||
| Service | `ExposeService`, `DeleteService` |
|
||||
| Provider | `CreateProvider`, `UpdateProvider`, `DeleteProvider`, `ConfigureProviderRefresh`, `DeleteProviderRefresh`, `RotateProviderCredential` |
|
||||
| Provider profile | `ImportProviderProfiles`, `UpdateProviderProfiles`, `DeleteProviderProfile` |
|
||||
| Policy and config | `UpdateConfig`, `ApproveDraftChunk`, `RejectDraftChunk`, `EditDraftChunk`, `UndoDraftChunk`, `ApproveAllDraftChunks`, `ClearDraftChunks` |
|
||||
|
||||
`UpdateSandbox` belongs to the separate `openshell.v1.SandboxUpdates` service
|
||||
on the same authenticated gateway listener. The other RPCs above remain on
|
||||
`openshell.v1.OpenShell`.
|
||||
|
||||
Use a nonzero, hyphenated UUID of exactly 36 characters. Uppercase and lowercase
|
||||
forms identify the same request. An empty ID preserves the existing behavior
|
||||
without deduplication. This ID is separate from transport correlation metadata.
|
||||
@@ -192,6 +196,35 @@ from this feature. Sandbox principals cannot opt into `UpdateConfig` admission.
|
||||
Check gateway compatibility
|
||||
before relying on IDs; older protobuf servers can silently ignore unknown fields.
|
||||
|
||||
### Sandbox image update recovery
|
||||
|
||||
`SandboxUpdates.UpdateSandbox` conditionally updates image, command, environment overrides, labels, and annotations for stopped
|
||||
Kubernetes sandboxes. It defaults to disabled and returns `FAILED_PRECONDITION`
|
||||
until the operator [activates a compatible deployment](/kubernetes/high-availability#activate-sandbox-updates).
|
||||
After activation, unsupported workload changes return `UNIMPLEMENTED`; metadata-only
|
||||
updates do not require driver support. For a new update, supply a named
|
||||
`workspace_scope`, sandbox `name`, at least one selected update field, and a required nonzero
|
||||
`expected_resource_version` read from `sandbox.metadata.resource_version`.
|
||||
A stale version returns `ABORTED`; read fresh state before constructing another
|
||||
conditional request. The operation does not start compute.
|
||||
|
||||
A failed or interrupted update can leave a pending operation. The desired spec
|
||||
alone does not prove driver reconciliation completed. Start, stop, and provisioning
|
||||
retry return `FAILED_PRECONDITION` until completion. Recovery runs automatically,
|
||||
including when new updates are disabled. To resume immediately, read the current
|
||||
resource version and send `UpdateSandbox` with no selected fields. The gateway
|
||||
loads the stored intent; no pending intent returns `FAILED_PRECONDITION`.
|
||||
Explicit delete can reclaim a sandbox whose update cannot complete.
|
||||
|
||||
If the original request supplied `request_id`, an error leaves its admission
|
||||
unresolved. Reusing that ID returns `REQUEST_OUTCOME_UNCERTAIN`; changing its
|
||||
payload returns `REQUEST_ID_PAYLOAD_MISMATCH`. For explicit resume, use the fresh
|
||||
version with a new ID or no ID. A confirmed success replays the current sandbox
|
||||
without applying another update.
|
||||
|
||||
See [Update the Image of a Stopped Sandbox](/how-it-works/sandboxes/overview#update-the-image-of-a-stopped-sandbox)
|
||||
for the stop, update, and start workflow and storage guarantees.
|
||||
|
||||
## Exec launch admission
|
||||
|
||||
`ExecSandbox` and `ExecSandboxInteractive` accept an optional `request_id` in
|
||||
|
||||
@@ -98,12 +98,22 @@ run_suite() {
|
||||
}
|
||||
|
||||
run_e2e() {
|
||||
if [ "${OPENSHELL_E2E_KUBE_TEST:-}" = "kubernetes_image_update" ]; then
|
||||
cargo test --locked --manifest-path "${ROOT}/tests/suites/features/Cargo.toml" \
|
||||
-p openshell-test-feature-kubernetes-updates -- --nocapture
|
||||
return
|
||||
fi
|
||||
run_conformance
|
||||
if [ "${E2E_FEATURES}" = "e2e-conformance-only" ]; then
|
||||
echo "::notice::Skipping Rust e2e tests (e2e-conformance-only)"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [ -z "${OPENSHELL_E2E_KUBE_TEST:-}" ]; then
|
||||
cargo test --locked --manifest-path "${ROOT}/tests/suites/features/Cargo.toml" \
|
||||
-p openshell-test-feature-kubernetes-updates -- --nocapture
|
||||
fi
|
||||
|
||||
# Pin `--target-dir` so the profile `junit.path` resolves regardless of `CARGO_TARGET_DIR`.
|
||||
cargo nextest run --profile e2e-kubernetes \
|
||||
--config-file "${ROOT}/.config/nextest.toml" \
|
||||
|
||||
@@ -1319,6 +1319,12 @@ helm_extra_args+=(--set supervisor.sandboxRuntime.networkPolicyEnforced=true)
|
||||
if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then
|
||||
helm_extra_args+=(--set "supervisor.network.additionalCaConfigMapName=${ADDITIONAL_CA_SOURCE_CONFIG_MAP}")
|
||||
fi
|
||||
# This wrapper installs an isolated, uniformly built test gateway, not a
|
||||
# production rolling upgrade. Explicitly activate the image-update scenario;
|
||||
# callers can keep activation disabled for negative coverage.
|
||||
if [ "${OPENSHELL_E2E_KUBE_IMAGE_UPDATES:-1}" = "1" ]; then
|
||||
helm_extra_args+=(--set 'gatewayConfig.openshell\.gateway.enable_sandbox_updates=true')
|
||||
fi
|
||||
if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then
|
||||
if [ "${OPENSHELL_E2E_KUBE_BUILD_IMAGES}" != "1" ]; then
|
||||
echo "ERROR: external Kubernetes driver e2e requires OPENSHELL_E2E_KUBE_BUILD_IMAGES=1." >&2
|
||||
|
||||
@@ -74,6 +74,35 @@ service ComputeDriver {
|
||||
rpc DeleteWorkspace(DeleteWorkspaceRequest) returns (DeleteWorkspaceResponse);
|
||||
}
|
||||
|
||||
// Optional stopped-sandbox configuration update extension on the same endpoint.
|
||||
// Baseline generated driver traits remain unchanged. Negotiate
|
||||
// openshell.compute.update-sandbox-configuration before calling this service.
|
||||
service ComputeDriverConfigurationUpdates {
|
||||
rpc UpdateSandboxConfiguration(UpdateSandboxConfigurationRequest)
|
||||
returns (UpdateSandboxConfigurationResponse);
|
||||
}
|
||||
|
||||
message UpdateSandboxConfigurationRequest {
|
||||
// Shared lifecycle, image, revision and runtime identity preconditions.
|
||||
SandboxUpdatePreconditions preconditions = 1;
|
||||
// Absent leaves the command unchanged; empty restores the default shell.
|
||||
SandboxMainProcessUpdate main_process = 2;
|
||||
// Effective user environment after template inheritance. Presence replaces it.
|
||||
SandboxEnvironmentUpdate environment = 3;
|
||||
}
|
||||
|
||||
// Only workload argv and its existing TTY setting are carried to the driver.
|
||||
message SandboxMainProcessUpdate {
|
||||
repeated string command = 1;
|
||||
bool tty = 2;
|
||||
}
|
||||
|
||||
message SandboxEnvironmentUpdate {
|
||||
map<string, string> values = 1;
|
||||
}
|
||||
|
||||
message UpdateSandboxConfigurationResponse {}
|
||||
|
||||
message GetCapabilitiesRequest {
|
||||
// Gateway protocol metadata. Drivers must reject unmet requirements.
|
||||
openshell.extension.v1.PeerMetadata gateway = 1;
|
||||
@@ -441,6 +470,26 @@ message CreateSandboxResponse {
|
||||
string runtime_identity = 1;
|
||||
}
|
||||
|
||||
message SandboxUpdatePreconditions {
|
||||
string sandbox_id = 1;
|
||||
string name = 2;
|
||||
string image = 3;
|
||||
// Pre-update image. A different current image conflicts unless durable driver
|
||||
// evidence identifies this exact operation as already applied.
|
||||
string expected_image = 4;
|
||||
// Never adopt a same-name replacement resource.
|
||||
string expected_runtime_identity = 5;
|
||||
// Stable UUID across retries, recorded atomically with the image change.
|
||||
string operation_id = 6;
|
||||
// Perform all admission and state checks without writing any resource. The
|
||||
// gateway uses this before recording durable intent; apply rechecks them.
|
||||
bool validate_only = 7;
|
||||
// Monotonic sandbox image revision. The driver atomically records it with
|
||||
// the image and rejects earlier revisions even if expected_image matches.
|
||||
uint64 revision = 8;
|
||||
}
|
||||
|
||||
|
||||
message StopSandboxRequest {
|
||||
// Stable sandbox ID stored by the gateway.
|
||||
string sandbox_id = 1;
|
||||
|
||||
@@ -858,6 +858,22 @@ service OpenShell {
|
||||
}
|
||||
}
|
||||
|
||||
// Optional sandbox mutations are separate from OpenShell so existing generated
|
||||
// OpenShell server implementations remain source-compatible.
|
||||
service SandboxUpdates {
|
||||
// Update the workload image, command, environment, labels, or annotations of a
|
||||
// stopped sandbox while preserving its identity and retained storage. Requires
|
||||
// compute-driver support for the selected changes and explicit gateway activation
|
||||
// of sandbox updates.
|
||||
rpc UpdateSandbox(UpdateSandboxRequest) returns (SandboxResponse) {
|
||||
option (openshell.options.v1.authorization) = {
|
||||
auth_mode: "bearer"
|
||||
scope: "sandbox:write"
|
||||
workspace_role: "user"
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// IssueSandboxToken request. Empty body; identity is established by the
|
||||
// authentication credentials carried in the request headers (a projected
|
||||
// Kubernetes ServiceAccount JWT in the K8s driver path).
|
||||
@@ -1330,6 +1346,40 @@ message CreateSandboxRequest {
|
||||
repeated SandboxServiceExposure service_exposures = 9;
|
||||
}
|
||||
|
||||
// Update selected configuration of a stopped sandbox. Omitted fields are unchanged.
|
||||
// Start explicitly after a successful update. With no fields selected, resume
|
||||
// the pending operation stored by the gateway using the current resource version.
|
||||
message UpdateSandboxRequest {
|
||||
// Workspace scope. Only a named workspace selection is accepted.
|
||||
openshell.datamodel.v1.WorkspaceSelector workspace_scope = 1;
|
||||
// Canonical name of the existing sandbox; this is not a rename or upsert.
|
||||
string name = 2;
|
||||
// Nonempty OCI workload image reference. This does not change trusted runtime
|
||||
// images or reseed an existing workspace.
|
||||
optional string image = 3;
|
||||
// Replace workload argv; an empty list restores image-resolved default shell behavior.
|
||||
SandboxCommandUpdate command = 6;
|
||||
// Replace sandbox-level overrides. Empty clears overrides; template defaults remain.
|
||||
SandboxMapUpdate environment = 7;
|
||||
// Replace user metadata, preserving gateway-owned keys. These do not set pod metadata.
|
||||
SandboxMapUpdate labels = 8;
|
||||
SandboxMapUpdate annotations = 9;
|
||||
// Required nonzero resource version read by the caller.
|
||||
uint64 expected_resource_version = 4;
|
||||
// Optional nonzero UUID for durable at-most-once admission.
|
||||
string request_id = 5;
|
||||
}
|
||||
|
||||
// Presence selects replacement, including an explicitly empty list.
|
||||
message SandboxCommandUpdate {
|
||||
repeated string argv = 1;
|
||||
}
|
||||
|
||||
// Presence selects replacement, including an explicitly empty map.
|
||||
message SandboxMapUpdate {
|
||||
map<string, string> values = 1;
|
||||
}
|
||||
|
||||
message CreateSandboxTemplateRequest {
|
||||
// Workspace scope. Only a named workspace selection is accepted.
|
||||
openshell.datamodel.v1.WorkspaceSelector workspace_scope = 2;
|
||||
|
||||
@@ -135,7 +135,9 @@ func TestConverterCoversAllProtoFields_SandboxStatus(t *testing.T) {
|
||||
// Provisioning carries gateway-owned attempt, deadline, and cleanup state;
|
||||
// the curated API exposes its outcome through phase and conditions. Detailed
|
||||
// lifecycle bookkeeping remains available through the raw protobuf API.
|
||||
skipped := fieldSet{"main_process_instance_id": true, "configuration_activated": true, "provisioning": true}
|
||||
// Image updates are raw-only; their pending reconciliation intent and
|
||||
// monotonic image revision bookkeeping are not part of the curated status model.
|
||||
skipped := fieldSet{"main_process_instance_id": true, "configuration_activated": true, "provisioning": true, "image_update": true, "image_revision": true}
|
||||
|
||||
assertAllFieldsCovered(t, (&pb.SandboxStatus{}).ProtoReflect().Descriptor(), handled, skipped)
|
||||
}
|
||||
|
||||
+1694
-1449
File diff suppressed because it is too large
Load Diff
@@ -3635,3 +3635,119 @@ var OpenShell_ServiceDesc = grpc.ServiceDesc{
|
||||
},
|
||||
Metadata: "openshell.proto",
|
||||
}
|
||||
|
||||
const (
|
||||
SandboxUpdates_UpdateSandbox_FullMethodName = "/openshell.v1.SandboxUpdates/UpdateSandbox"
|
||||
)
|
||||
|
||||
// SandboxUpdatesClient is the client API for SandboxUpdates service.
|
||||
//
|
||||
// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
|
||||
//
|
||||
// Optional sandbox mutations are separate from OpenShell so existing generated
|
||||
// OpenShell server implementations remain source-compatible.
|
||||
type SandboxUpdatesClient interface {
|
||||
// Update the workload image, command, environment, labels, or annotations of a
|
||||
// stopped sandbox while preserving its identity and retained storage. Requires
|
||||
// compute-driver support for the selected changes and explicit gateway activation
|
||||
// of sandbox updates.
|
||||
UpdateSandbox(ctx context.Context, in *UpdateSandboxRequest, opts ...grpc.CallOption) (*SandboxResponse, error)
|
||||
}
|
||||
|
||||
type sandboxUpdatesClient struct {
|
||||
cc grpc.ClientConnInterface
|
||||
}
|
||||
|
||||
func NewSandboxUpdatesClient(cc grpc.ClientConnInterface) SandboxUpdatesClient {
|
||||
return &sandboxUpdatesClient{cc}
|
||||
}
|
||||
|
||||
func (c *sandboxUpdatesClient) UpdateSandbox(ctx context.Context, in *UpdateSandboxRequest, opts ...grpc.CallOption) (*SandboxResponse, error) {
|
||||
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
|
||||
out := new(SandboxResponse)
|
||||
err := c.cc.Invoke(ctx, SandboxUpdates_UpdateSandbox_FullMethodName, in, out, cOpts...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// SandboxUpdatesServer is the server API for SandboxUpdates service.
|
||||
// All implementations must embed UnimplementedSandboxUpdatesServer
|
||||
// for forward compatibility.
|
||||
//
|
||||
// Optional sandbox mutations are separate from OpenShell so existing generated
|
||||
// OpenShell server implementations remain source-compatible.
|
||||
type SandboxUpdatesServer interface {
|
||||
// Update the workload image, command, environment, labels, or annotations of a
|
||||
// stopped sandbox while preserving its identity and retained storage. Requires
|
||||
// compute-driver support for the selected changes and explicit gateway activation
|
||||
// of sandbox updates.
|
||||
UpdateSandbox(context.Context, *UpdateSandboxRequest) (*SandboxResponse, error)
|
||||
mustEmbedUnimplementedSandboxUpdatesServer()
|
||||
}
|
||||
|
||||
// UnimplementedSandboxUpdatesServer must be embedded to have
|
||||
// forward compatible implementations.
|
||||
//
|
||||
// NOTE: this should be embedded by value instead of pointer to avoid a nil
|
||||
// pointer dereference when methods are called.
|
||||
type UnimplementedSandboxUpdatesServer struct{}
|
||||
|
||||
func (UnimplementedSandboxUpdatesServer) UpdateSandbox(context.Context, *UpdateSandboxRequest) (*SandboxResponse, error) {
|
||||
return nil, status.Error(codes.Unimplemented, "method UpdateSandbox not implemented")
|
||||
}
|
||||
func (UnimplementedSandboxUpdatesServer) mustEmbedUnimplementedSandboxUpdatesServer() {}
|
||||
func (UnimplementedSandboxUpdatesServer) testEmbeddedByValue() {}
|
||||
|
||||
// UnsafeSandboxUpdatesServer may be embedded to opt out of forward compatibility for this service.
|
||||
// Use of this interface is not recommended, as added methods to SandboxUpdatesServer will
|
||||
// result in compilation errors.
|
||||
type UnsafeSandboxUpdatesServer interface {
|
||||
mustEmbedUnimplementedSandboxUpdatesServer()
|
||||
}
|
||||
|
||||
func RegisterSandboxUpdatesServer(s grpc.ServiceRegistrar, srv SandboxUpdatesServer) {
|
||||
// If the following call panics, it indicates UnimplementedSandboxUpdatesServer was
|
||||
// embedded by pointer and is nil. This will cause panics if an
|
||||
// unimplemented method is ever invoked, so we test this at initialization
|
||||
// time to prevent it from happening at runtime later due to I/O.
|
||||
if t, ok := srv.(interface{ testEmbeddedByValue() }); ok {
|
||||
t.testEmbeddedByValue()
|
||||
}
|
||||
s.RegisterService(&SandboxUpdates_ServiceDesc, srv)
|
||||
}
|
||||
|
||||
func _SandboxUpdates_UpdateSandbox_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
|
||||
in := new(UpdateSandboxRequest)
|
||||
if err := dec(in); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if interceptor == nil {
|
||||
return srv.(SandboxUpdatesServer).UpdateSandbox(ctx, in)
|
||||
}
|
||||
info := &grpc.UnaryServerInfo{
|
||||
Server: srv,
|
||||
FullMethod: SandboxUpdates_UpdateSandbox_FullMethodName,
|
||||
}
|
||||
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
|
||||
return srv.(SandboxUpdatesServer).UpdateSandbox(ctx, req.(*UpdateSandboxRequest))
|
||||
}
|
||||
return interceptor(ctx, in, info, handler)
|
||||
}
|
||||
|
||||
// SandboxUpdates_ServiceDesc is the grpc.ServiceDesc for SandboxUpdates service.
|
||||
// It's only intended for direct use with grpc.RegisterService,
|
||||
// and not to be introspected or modified (even as a copy)
|
||||
var SandboxUpdates_ServiceDesc = grpc.ServiceDesc{
|
||||
ServiceName: "openshell.v1.SandboxUpdates",
|
||||
HandlerType: (*SandboxUpdatesServer)(nil),
|
||||
Methods: []grpc.MethodDesc{
|
||||
{
|
||||
MethodName: "UpdateSandbox",
|
||||
Handler: _SandboxUpdates_UpdateSandbox_Handler,
|
||||
},
|
||||
},
|
||||
Streams: []grpc.StreamDesc{},
|
||||
Metadata: "openshell.proto",
|
||||
}
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
package compatibility_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net"
|
||||
"testing"
|
||||
|
||||
dm "github.com/NVIDIA/OpenShell/sdk/go/proto/datamodelv1"
|
||||
pb "github.com/NVIDIA/OpenShell/sdk/go/proto/openshellv1"
|
||||
"github.com/stretchr/testify/require"
|
||||
"google.golang.org/grpc"
|
||||
"google.golang.org/grpc/codes"
|
||||
"google.golang.org/grpc/credentials/insecure"
|
||||
"google.golang.org/grpc/metadata"
|
||||
"google.golang.org/grpc/status"
|
||||
"google.golang.org/grpc/test/bufconn"
|
||||
"google.golang.org/protobuf/proto"
|
||||
"google.golang.org/protobuf/reflect/protoreflect"
|
||||
)
|
||||
|
||||
func TestSandboxStatusRetainsBaselinePublicShape(t *testing.T) {
|
||||
// Freeze the pre-image-update fields and tags: private operation state must
|
||||
// not enlarge the public Rust struct or leak into generated SDK messages.
|
||||
fields := []string{
|
||||
"agent_pod", "agent_fd", "sandbox_fd", "conditions", "phase",
|
||||
"current_policy_version", "main_process_instance_id", "exit_code",
|
||||
"endpoint_statuses", "configuration_admission", "configuration_activated",
|
||||
"provisioning", "restart_count", "next_restart_time",
|
||||
"main_process_started_time", "last_token_grant_failure",
|
||||
}
|
||||
descriptor := (&pb.SandboxStatus{}).ProtoReflect().Descriptor()
|
||||
require.Equal(t, len(fields), descriptor.Fields().Len())
|
||||
for i, name := range fields {
|
||||
field := descriptor.Fields().ByName(protoreflect.Name(name))
|
||||
require.NotNil(t, field, name)
|
||||
require.Equal(t, protoreflect.FieldNumber(i+2), field.Number(), name)
|
||||
}
|
||||
require.Nil(t, pb.File_openshell_proto.Messages().ByName("SandboxImageUpdate"))
|
||||
legacy := pb.File_openshell_proto.Services().ByName("OpenShell")
|
||||
require.Nil(t, legacy.Methods().ByName("UpdateSandbox"))
|
||||
updates := pb.File_openshell_proto.Services().ByName("SandboxUpdates")
|
||||
require.NotNil(t, updates)
|
||||
require.Equal(t, 1, updates.Methods().Len())
|
||||
method := updates.Methods().Get(0)
|
||||
require.Equal(t, protoreflect.FullName("openshell.v1.UpdateSandboxRequest"), method.Input().FullName())
|
||||
require.Equal(t, protoreflect.FullName("openshell.v1.SandboxResponse"), method.Output().FullName())
|
||||
require.Equal(t, "/openshell.v1.SandboxUpdates/UpdateSandbox", pb.SandboxUpdates_UpdateSandbox_FullMethodName)
|
||||
}
|
||||
|
||||
type legacyHealthServer struct {
|
||||
pb.UnimplementedOpenShellServer
|
||||
}
|
||||
|
||||
func (legacyHealthServer) Health(context.Context, *pb.HealthRequest) (*pb.HealthResponse, error) {
|
||||
return &pb.HealthResponse{Version: "legacy-compatible"}, nil
|
||||
}
|
||||
|
||||
type imageUpdateServer struct {
|
||||
pb.UnimplementedSandboxUpdatesServer
|
||||
request *pb.UpdateSandboxRequest
|
||||
}
|
||||
|
||||
func (s *imageUpdateServer) UpdateSandbox(_ context.Context, request *pb.UpdateSandboxRequest) (*pb.SandboxResponse, error) {
|
||||
s.request = proto.Clone(request).(*pb.UpdateSandboxRequest)
|
||||
return &pb.SandboxResponse{Sandbox: &pb.Sandbox{
|
||||
Metadata: &dm.ObjectMeta{Id: "stable-id", Name: request.Name, Workspace: request.WorkspaceScope.GetWorkspace()},
|
||||
Spec: &pb.SandboxSpec{Template: &pb.SandboxTemplate{Image: request.GetImage()}},
|
||||
}}, nil
|
||||
}
|
||||
|
||||
func TestSeparateSandboxUpdatesTransportCoexistsWithLegacyOpenShell(t *testing.T) {
|
||||
listener := bufconn.Listen(1024 * 1024)
|
||||
server := grpc.NewServer(grpc.UnaryInterceptor(func(ctx context.Context, request any, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (any, error) {
|
||||
if info.FullMethod == pb.SandboxUpdates_UpdateSandbox_FullMethodName {
|
||||
md, _ := metadata.FromIncomingContext(ctx)
|
||||
if len(md.Get("authorization")) == 0 {
|
||||
return nil, status.Error(codes.Unauthenticated, "bearer required")
|
||||
}
|
||||
}
|
||||
return handler(ctx, request)
|
||||
}))
|
||||
updatesServer := &imageUpdateServer{}
|
||||
pb.RegisterOpenShellServer(server, legacyHealthServer{})
|
||||
pb.RegisterSandboxUpdatesServer(server, updatesServer)
|
||||
go func() { _ = server.Serve(listener) }()
|
||||
t.Cleanup(server.Stop)
|
||||
t.Cleanup(func() { _ = listener.Close() })
|
||||
conn, err := grpc.NewClient("passthrough:///sandbox-updates", grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithContextDialer(func(context.Context, string) (net.Conn, error) {
|
||||
return listener.Dial()
|
||||
}))
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = conn.Close() })
|
||||
legacy := pb.NewOpenShellClient(conn)
|
||||
health, err := legacy.Health(context.Background(), &pb.HealthRequest{})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "legacy-compatible", health.Version)
|
||||
updates := pb.NewSandboxUpdatesClient(conn)
|
||||
request := &pb.UpdateSandboxRequest{
|
||||
WorkspaceScope: &dm.WorkspaceSelector{Selection: &dm.WorkspaceSelector_Workspace{Workspace: "team"}},
|
||||
Name: "sandbox",
|
||||
Image: proto.String("example/workload:new"),
|
||||
ExpectedResourceVersion: 42,
|
||||
RequestId: "550e8400-e29b-41d4-a716-446655440000",
|
||||
}
|
||||
_, err = updates.UpdateSandbox(context.Background(), request)
|
||||
require.Equal(t, codes.Unauthenticated, status.Code(err))
|
||||
require.Nil(t, updatesServer.request)
|
||||
ctx := metadata.NewOutgoingContext(context.Background(), metadata.Pairs("authorization", "Bearer test-token"))
|
||||
response, err := updates.UpdateSandbox(ctx, request)
|
||||
require.NoError(t, err)
|
||||
require.True(t, proto.Equal(request, updatesServer.request))
|
||||
require.Equal(t, "stable-id", response.Sandbox.Metadata.Id)
|
||||
require.Equal(t, "team", response.Sandbox.Metadata.Workspace)
|
||||
require.Equal(t, request.GetImage(), response.Sandbox.Spec.Template.Image)
|
||||
// The old service has no accidental update endpoint.
|
||||
err = conn.Invoke(ctx, "/openshell.v1.OpenShell/UpdateSandbox", request, &pb.SandboxResponse{})
|
||||
require.Equal(t, codes.Unimplemented, status.Code(err))
|
||||
}
|
||||
@@ -3,10 +3,55 @@
|
||||
|
||||
import type { MessageInitShape } from '@bufbuild/protobuf';
|
||||
import { timestampFromDate } from '@bufbuild/protobuf/wkt';
|
||||
import { createRouterTransport } from '@connectrpc/connect';
|
||||
import { createClient, createRouterTransport } from '@connectrpc/connect';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { SandboxClient } from './index.js';
|
||||
import { EndpointResult, type EndpointStatusSchema, OpenShell, SandboxPhase } from './raw.js';
|
||||
import { EndpointResult, type EndpointStatusSchema, OpenShell, SandboxPhase, SandboxUpdates } from './raw.js';
|
||||
|
||||
describe('raw sandbox image updates', () => {
|
||||
it.each([true, false])('uses a separate service on the existing transport (binary: %s)', async (useBinaryFormat) => {
|
||||
const transport = createRouterTransport(
|
||||
(router) => {
|
||||
router.service(OpenShell, {
|
||||
getSandbox: () => ({ sandbox: { metadata: { id: 'stable-id' } } }),
|
||||
});
|
||||
router.service(SandboxUpdates, {
|
||||
updateSandbox: (request) => {
|
||||
expect(request.workspaceScope?.selection).toEqual({ case: 'workspace', value: 'team' });
|
||||
expect(request.name).toBe('sandbox');
|
||||
expect(request.expectedResourceVersion).toBe(42n);
|
||||
expect(request.requestId).toBe('550e8400-e29b-41d4-a716-446655440000');
|
||||
return {
|
||||
sandbox: {
|
||||
metadata: { id: 'stable-id', name: request.name, workspace: 'team' },
|
||||
spec: { template: { image: request.image } },
|
||||
status: { phase: SandboxPhase.STOPPED },
|
||||
},
|
||||
};
|
||||
},
|
||||
});
|
||||
},
|
||||
{ transport: { useBinaryFormat } },
|
||||
);
|
||||
const legacy = new SandboxClient(transport);
|
||||
const updates = createClient(SandboxUpdates, legacy.transport);
|
||||
expect(Object.hasOwn(OpenShell.method, 'updateSandbox')).toBe(false);
|
||||
expect(SandboxUpdates.typeName).toBe('openshell.v1.SandboxUpdates');
|
||||
const before = await legacy.raw.getSandbox({ name: 'sandbox' });
|
||||
const response = await updates.updateSandbox({
|
||||
workspaceScope: { selection: { case: 'workspace', value: 'team' } },
|
||||
name: 'sandbox',
|
||||
image: 'example/workload:new',
|
||||
expectedResourceVersion: 42n,
|
||||
requestId: '550e8400-e29b-41d4-a716-446655440000',
|
||||
});
|
||||
expect(response.sandbox?.metadata?.id).toBe(before.sandbox?.metadata?.id);
|
||||
expect(response.sandbox?.spec?.template?.image).toBe('example/workload:new');
|
||||
expect(response.sandbox?.status?.phase).toBe(SandboxPhase.STOPPED);
|
||||
expect(response.sandbox?.status).not.toHaveProperty('imageUpdate');
|
||||
expect(response.sandbox?.status).not.toHaveProperty('imageRevision');
|
||||
});
|
||||
});
|
||||
|
||||
describe('raw sandbox endpoint status', () => {
|
||||
it.each([true, false])(
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
export * from './gen/datamodel_pb.js';
|
||||
export * from './gen/google/rpc/error_details_pb.js';
|
||||
// Advanced surface: the full generated protobuf types (messages, enums, and the
|
||||
// OpenShell service descriptor) for callers using the raw escape hatch on
|
||||
// OpenShell and SandboxUpdates service descriptors) for callers using the raw escape hatch on
|
||||
// OpenShellClient / SandboxClient (`.raw` and `.transport`). These are the
|
||||
// uncurated wire types; import them from '@nvidia/openshell-sdk/raw'. The
|
||||
// curated entry point stays free of generated types so its surface does not
|
||||
|
||||
@@ -1435,6 +1435,52 @@ network supervisor logs for CONNECT and upstream-proxy decisions:
|
||||
kubectl -n <sandbox-namespace> logs pod/<supervisor-pod> --tail=200 | grep -Ei 'upstream|connect|proxy'
|
||||
```
|
||||
|
||||
#### Sandbox image-update activation and recovery
|
||||
|
||||
`openshell.v1.SandboxUpdates/UpdateSandbox` uses the existing authenticated
|
||||
gateway listener and defaults to disabled. Check the effective
|
||||
`[openshell.gateway] enable_sandbox_updates` value and its overrides,
|
||||
`--enable-sandbox-updates true` or
|
||||
`OPENSHELL_ENABLE_SANDBOX_UPDATES=true`. Canonical Helm values use:
|
||||
|
||||
```yaml
|
||||
gatewayConfig:
|
||||
"openshell.gateway":
|
||||
enable_sandbox_updates: true
|
||||
```
|
||||
|
||||
Do not enable the flag to work around an error. It attests that all incompatible
|
||||
gateways, including unregistered processes and Jobs, have been drained and
|
||||
retired, their outstanding driver operations have settled, and old binaries
|
||||
cannot return through rollout or rollback configurations. Keep it false during
|
||||
a rolling upgrade. SQLite supports one gateway process, not shared-file HA.
|
||||
|
||||
On PostgreSQL, fresh gateway membership must advertise sandbox-update protocol 1;
|
||||
malformed or truncated membership reads fail closed. Administrative, monitoring,
|
||||
and backup connections do not block updates, and custom `application_name` values
|
||||
are preserved. Membership expiry does not prove a gateway has stopped. Follow the
|
||||
[activation checklist](https://docs.nvidia.com/openshell/latest/kubernetes/high-availability#activate-sandbox-updates).
|
||||
|
||||
The gateway retries accepted updates automatically, even with new updates disabled.
|
||||
Repeated automatic failures back off up to 15 minutes per gateway. Replicas retry
|
||||
independently, and restarting a gateway resets its delay. Recovery warnings include
|
||||
the attempt count and retry delay.
|
||||
To resume immediately, read the sandbox resource version and call `UpdateSandbox`
|
||||
with that version and no selected fields. Restore a compatible driver if recovery
|
||||
reports missing support. Start, stop, and provisioning retry retain the barrier;
|
||||
explicit deletion can reclaim an unrecoverable sandbox and its storage. Preserve
|
||||
request admission semantics: use a new ID or no ID for the explicit resume request,
|
||||
and never change the original payload under its original ID. See the
|
||||
[recovery contract](https://docs.nvidia.com/openshell/latest/sdk/api-errors#sandbox-image-update-recovery).
|
||||
|
||||
Split Helm deployments need the current `openshell-workspace` Role. Update
|
||||
preflight requires `get` on Pods: it checks the controller-recorded workload
|
||||
name, the Sandbox resource name, and `os-supervisor-<sandbox-id>` directly.
|
||||
Any remaining Pod at those names, including a terminating Pod, blocks the update.
|
||||
The checks make at most three Pod reads regardless of namespace size and do not
|
||||
discover unexpectedly named Pods. Upgrade the workspace chart alongside the
|
||||
gateway chart.
|
||||
|
||||
### Step 7: Check VM-Backed Gateways
|
||||
|
||||
Use the VM driver logs and host diagnostics available in the user's environment. Verify:
|
||||
|
||||
@@ -527,6 +527,12 @@ stopped or completed. Starting a retained `Completed` or
|
||||
invalidates SSH sessions from the previous runtime generation. Delete remains
|
||||
the operation that removes retained state.
|
||||
|
||||
Before updating a retained `Completed` or `Error/MainProcessFailed` sandbox,
|
||||
run `sandbox stop`. It cleans up compute and transitions to `Stopped`, retaining
|
||||
the exit code until the next start. If the terminal sandbox's compute resource
|
||||
is already gone, stop succeeds and preserves its terminal phase and exit code;
|
||||
configuration updates require retained compute.
|
||||
|
||||
The sandbox's SSH host identity survives stop/start and runtime restarts.
|
||||
API clients can read its public fingerprint from the sandbox resource or
|
||||
SSH-session response and pin it to the sandbox ID. Recreating a deleted
|
||||
@@ -554,6 +560,53 @@ state cannot be restored; use `sandbox start` for an explicit clean boot. `sandb
|
||||
`sandbox stop`, and `sandbox delete` on a suspended sandbox discard the
|
||||
retained memory state.
|
||||
|
||||
### Change a stopped sandbox's workload image
|
||||
|
||||
Do not invent a `sandbox update` command or use delete/create to change an
|
||||
existing sandbox's configuration. The separate
|
||||
`openshell.v1.SandboxUpdates/UpdateSandbox` RPC uses the same gateway listener
|
||||
and authentication as `OpenShell`. It defaults to disabled; the operator must
|
||||
complete the published
|
||||
[activation checklist](https://docs.nvidia.com/openshell/latest/kubernetes/high-availability#activate-sandbox-updates)
|
||||
before enabling `enable_sandbox_updates`. Keep it false during a rolling
|
||||
upgrade and never enable it merely to bypass a failed compatibility check.
|
||||
After activation, image, command, and environment updates initially support only
|
||||
stopped Kubernetes sandboxes; other drivers return `UNIMPLEMENTED` for those
|
||||
fields. Metadata-only updates also support stopped sandboxes on other drivers. Stop and
|
||||
wait for `Stopped`, read the sandbox's current resource version through
|
||||
`GetSandbox`, call raw `UpdateSandbox` with the named workspace, sandbox name,
|
||||
selected image, command, environment overrides, labels or annotations, and required nonzero `expected_resource_version`, then start after
|
||||
success. Rust SDK callers use `client.raw_sandbox_updates_fresh()`; generated
|
||||
clients use `SandboxUpdates`, not the unchanged `OpenShell` service. There is no
|
||||
curated SDK update method. The sandbox identity, Kubernetes
|
||||
custom resource, and persistent volume claim remain; the existing workspace is
|
||||
not reseeded and trusted runtime images do not change. Omitted update fields are
|
||||
unchanged; supplied lists and maps replace their user-owned values, including
|
||||
explicitly empty values. Template environment defaults still apply. System metadata
|
||||
is protected; top-level labels and annotations do not update pod metadata.
|
||||
Combined configuration changes require the configuration-update driver capability
|
||||
and a uniformly upgraded gateway deployment. Metadata-only updates need no
|
||||
driver capability and preserve the retention annotation.
|
||||
|
||||
For an image that fails to start and reaches a provisioning timeout, wait for
|
||||
cleanup completion and settlement of the original driver operation, then run
|
||||
`sandbox stop` to obtain `Stopped` before correcting the image. This retains the
|
||||
timeout diagnostic and storage and does not bypass pending cleanup.
|
||||
|
||||
An operation-pending error blocks start, stop, and provisioning retry. The gateway
|
||||
limits preflight and each accepted update attempt to one minute each. A timeout
|
||||
after acceptance preserves pending intent for recovery; it does not imply rollback.
|
||||
The gateway retries automatically even when new updates are disabled. Repeated automatic
|
||||
failures back off up to 15 minutes per gateway; restarting a gateway resets the delay.
|
||||
To resume immediately,
|
||||
read a fresh resource version and call `UpdateSandbox` with no selected fields;
|
||||
the gateway loads the stored targets. Use a new request ID or no ID for this
|
||||
explicit resume. An unresolved original ID returns uncertainty, and its payload
|
||||
must not change. Explicit delete can reclaim a sandbox whose update cannot
|
||||
complete, including its storage. See the published
|
||||
[image update workflow](https://docs.nvidia.com/openshell/latest/how-it-works/sandboxes/overview#update-the-image-of-a-stopped-sandbox)
|
||||
and [API errors](https://docs.nvidia.com/openshell/latest/sdk/api-errors).
|
||||
|
||||
---
|
||||
|
||||
## Workflow 4: Policy Iteration Loop
|
||||
|
||||
@@ -254,6 +254,14 @@ run = [
|
||||
"AGENT_SANDBOX_VERSION=v0.4.6 e2e/rust/e2e-kubernetes.sh",
|
||||
]
|
||||
|
||||
["e2e:kubernetes:updates"]
|
||||
description = "Validate sandbox updates with the combined Helm chart on both Agent Sandbox APIs"
|
||||
env = { OPENSHELL_E2E_KUBE_TEST = "kubernetes_image_update" }
|
||||
run = [
|
||||
"e2e/rust/e2e-kubernetes.sh",
|
||||
"AGENT_SANDBOX_VERSION=v0.4.6 e2e/rust/e2e-kubernetes.sh",
|
||||
]
|
||||
|
||||
["e2e:kubernetes:isolation"]
|
||||
description = "Run Kubernetes e2e with the workload network fence and separate supervisor"
|
||||
run = "e2e/rust/e2e-kubernetes.sh"
|
||||
|
||||
Generated
+753
-47
File diff suppressed because it is too large
Load Diff
@@ -3,4 +3,4 @@
|
||||
|
||||
[workspace]
|
||||
resolver = "2"
|
||||
members = ["oci-image", "provider-refresh/keycloak"]
|
||||
members = ["oci-image", "kubernetes-updates", "provider-refresh/keycloak"]
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
[package]
|
||||
name = "openshell-test-feature-kubernetes-updates"
|
||||
version = "0.0.0"
|
||||
edition = "2024"
|
||||
|
||||
[dependencies]
|
||||
openshell-testkit = { path = "../../../testkit/testkit" }
|
||||
serde_json = "1"
|
||||
tempfile = "3"
|
||||
tokio = { version = "1.43", features = ["process", "time"] }
|
||||
tonic = "0.14"
|
||||
uuid = { version = "1.10", features = ["v4"] }
|
||||
@@ -0,0 +1,695 @@
|
||||
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
//! Exercise the public image-update RPC in both Agent Sandbox API-version lanes.
|
||||
//! Kubernetes checks are scoped to test-owned resources.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::time::Duration;
|
||||
|
||||
use serde_json::Value;
|
||||
use std::sync::LazyLock;
|
||||
use testkit::sdk::raw::{AuthedGrpcClient, AuthedSandboxUpdatesClient, proto};
|
||||
use testkit::{Context, Sandbox};
|
||||
|
||||
// Multi-platform release manifests; no test-local tags or cluster image loads.
|
||||
const ORIGINAL_IMAGE_DEFAULT: &str = "mirror.gcr.io/library/alpine:3.21@sha256:ce64758a109eb420d874a118f87920e625e12d3634e03b4a5573fd9f6e5d3507";
|
||||
const UPDATED_IMAGE_DEFAULT: &str = "mirror.gcr.io/library/alpine:3.22@sha256:5291449c3df73caf6ed85e649dec1b9e818b39a5d8c871e97afc13e9cd5e8fa8";
|
||||
const SENTINEL_PATH: &str = "/sandbox/.image-update-sentinel";
|
||||
|
||||
// CI can use a private mirror without changing the tested image versions.
|
||||
static ORIGINAL_IMAGE: LazyLock<String> = LazyLock::new(|| {
|
||||
std::env::var("OPENSHELL_E2E_UPDATE_ORIGINAL_IMAGE")
|
||||
.unwrap_or_else(|_| ORIGINAL_IMAGE_DEFAULT.into())
|
||||
});
|
||||
static UPDATED_IMAGE: LazyLock<String> = LazyLock::new(|| {
|
||||
std::env::var("OPENSHELL_E2E_UPDATE_TARGET_IMAGE")
|
||||
.unwrap_or_else(|_| UPDATED_IMAGE_DEFAULT.into())
|
||||
});
|
||||
|
||||
fn request_id() -> String {
|
||||
uuid::Uuid::new_v4().to_string()
|
||||
}
|
||||
|
||||
fn rpc<T>(body: T) -> tonic::Request<T> {
|
||||
let mut request = tonic::Request::new(body);
|
||||
request.set_timeout(Duration::from_secs(60));
|
||||
request
|
||||
}
|
||||
|
||||
async fn get_sandbox(client: &mut AuthedGrpcClient, workspace: &str, name: &str) -> proto::Sandbox {
|
||||
client
|
||||
.get_sandbox(rpc(proto::GetSandboxRequest {
|
||||
workspace_scope: Some(proto::workspace_selector(workspace)),
|
||||
name: name.to_owned(),
|
||||
}))
|
||||
.await
|
||||
.expect("get test sandbox")
|
||||
.into_inner()
|
||||
.sandbox
|
||||
.expect("Sandbox response")
|
||||
}
|
||||
|
||||
/// Stop can still be followed by driver-watch status writes. Refresh the CAS
|
||||
/// version on a definite conflict, without retrying ambiguous driver failures.
|
||||
async fn update_stopped_sandbox(
|
||||
client: &mut AuthedGrpcClient,
|
||||
updates: &mut AuthedSandboxUpdatesClient,
|
||||
workspace: &str,
|
||||
mut request: proto::UpdateSandboxRequest,
|
||||
) -> (proto::Sandbox, proto::UpdateSandboxRequest) {
|
||||
let mut sandbox_id = None;
|
||||
for attempt in 0..20 {
|
||||
let current = get_sandbox(client, workspace, &request.name).await;
|
||||
let metadata = current.metadata.as_ref().expect("sandbox metadata");
|
||||
let id = sandbox_id.get_or_insert_with(|| metadata.id.clone());
|
||||
assert_eq!(
|
||||
&metadata.id, id,
|
||||
"conditional retry must not target a replacement"
|
||||
);
|
||||
let status = current.status.as_ref().expect("sandbox status");
|
||||
assert_eq!(status.phase, proto::SandboxPhase::Stopped as i32);
|
||||
assert!(
|
||||
status
|
||||
.provisioning
|
||||
.as_ref()
|
||||
.is_none_or(|record| !record.driver_operation_pending),
|
||||
"do not retry a newly selected update over a pending operation"
|
||||
);
|
||||
request.expected_resource_version = metadata.resource_version;
|
||||
if attempt > 0 && !request.request_id.is_empty() {
|
||||
// A changed version is a new request payload, not a replay.
|
||||
request.request_id = request_id();
|
||||
}
|
||||
match updates.update_sandbox(rpc(request.clone())).await {
|
||||
Ok(response) => {
|
||||
return (
|
||||
response.into_inner().sandbox.expect("updated Sandbox"),
|
||||
request,
|
||||
);
|
||||
}
|
||||
Err(error) => {
|
||||
let conflict = error.code() == tonic::Code::Aborted
|
||||
&& testkit::sdk::SdkError::from_status(error.clone())
|
||||
.error_details()
|
||||
.is_some_and(|details| {
|
||||
details.error_info().is_some_and(|info| {
|
||||
info.domain == "openshell.nvidia.com"
|
||||
&& info.reason == "RESOURCE_VERSION_CONFLICT"
|
||||
})
|
||||
});
|
||||
assert!(
|
||||
conflict && attempt < 19,
|
||||
"update stopped sandbox: {error:?}"
|
||||
);
|
||||
eprintln!(
|
||||
"refreshing sandbox {} after version conflict: {error}",
|
||||
request.name
|
||||
);
|
||||
tokio::time::sleep(Duration::from_millis(100)).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
unreachable!("final conflict fails the test")
|
||||
}
|
||||
|
||||
async fn kubectl_mutation(args: &[&str]) -> std::process::Output {
|
||||
let namespace = std::env::var("OPENSHELL_E2E_SANDBOX_NAMESPACE").expect("test namespace");
|
||||
let context = std::env::var("OPENSHELL_E2E_KUBE_CONTEXT_ACTIVE").expect("test context");
|
||||
tokio::process::Command::new("kubectl")
|
||||
.args([
|
||||
"--context",
|
||||
&context,
|
||||
"--namespace",
|
||||
&namespace,
|
||||
"--request-timeout=30s",
|
||||
])
|
||||
.args(args)
|
||||
.kill_on_drop(true)
|
||||
.output()
|
||||
.await
|
||||
.expect("kubectl mutation")
|
||||
}
|
||||
|
||||
/// Exercise real API-server immutable UID validation independently of the
|
||||
/// resourceVersion condition, after deleting and recreating the named resource.
|
||||
async fn assert_recreated_resource_rejects_stale_patch(template: &Value, name: &str) {
|
||||
use std::io::Write as _;
|
||||
let resource = serde_json::json!({
|
||||
"apiVersion": template["apiVersion"], "kind": "Sandbox",
|
||||
"metadata": {"name": name}, "spec": template["spec"],
|
||||
});
|
||||
let mut file = tempfile::NamedTempFile::new().expect("test manifest");
|
||||
file.write_all(serde_json::to_string(&resource).unwrap().as_bytes())
|
||||
.unwrap();
|
||||
let path = file.path().to_str().unwrap();
|
||||
let create = kubectl_mutation(&["create", "-f", path, "-o", "json"]).await;
|
||||
assert!(
|
||||
create.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&create.stderr)
|
||||
);
|
||||
let original: Value = serde_json::from_slice(&create.stdout).unwrap();
|
||||
let delete = kubectl_mutation(&[
|
||||
"delete",
|
||||
"sandboxes.agents.x-k8s.io",
|
||||
name,
|
||||
"--wait=true",
|
||||
"--timeout=60s",
|
||||
])
|
||||
.await;
|
||||
assert!(
|
||||
delete.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&delete.stderr)
|
||||
);
|
||||
let create = kubectl_mutation(&["create", "-f", path, "-o", "json"]).await;
|
||||
assert!(
|
||||
create.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&create.stderr)
|
||||
);
|
||||
let replacement: Value = serde_json::from_slice(&create.stdout).unwrap();
|
||||
let patch = serde_json::json!({"metadata": {
|
||||
"uid": original["metadata"]["uid"],
|
||||
"annotations": {"example.com/stale-patch": "must-not-apply"},
|
||||
}})
|
||||
.to_string();
|
||||
let rejected = kubectl_mutation(&[
|
||||
"patch",
|
||||
"sandboxes.agents.x-k8s.io",
|
||||
name,
|
||||
"--type=merge",
|
||||
"-p",
|
||||
&patch,
|
||||
])
|
||||
.await;
|
||||
let after = kubectl_get("sandboxes.agents.x-k8s.io", &[name]).await;
|
||||
let cleanup = kubectl_mutation(&[
|
||||
"delete",
|
||||
"sandboxes.agents.x-k8s.io",
|
||||
name,
|
||||
"--wait=true",
|
||||
"--timeout=60s",
|
||||
])
|
||||
.await;
|
||||
assert!(
|
||||
cleanup.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&cleanup.stderr)
|
||||
);
|
||||
assert_ne!(original["metadata"]["uid"], replacement["metadata"]["uid"]);
|
||||
assert!(!rejected.status.success(), "stale UID patch must fail");
|
||||
assert!(
|
||||
String::from_utf8_lossy(&rejected.stderr).contains("metadata.uid"),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&rejected.stderr)
|
||||
);
|
||||
assert!(after["metadata"]["annotations"]["example.com/stale-patch"].is_null());
|
||||
}
|
||||
|
||||
async fn kubectl_get(resource: &str, selector: &[&str]) -> Value {
|
||||
let namespace = std::env::var("OPENSHELL_E2E_SANDBOX_NAMESPACE")
|
||||
.expect("Kubernetes E2E wrapper must provide the sandbox namespace");
|
||||
let context = std::env::var("OPENSHELL_E2E_KUBE_CONTEXT_ACTIVE")
|
||||
.expect("Kubernetes E2E wrapper must provide its active context");
|
||||
assert!(!namespace.trim().is_empty() && !context.trim().is_empty());
|
||||
let output = tokio::process::Command::new("kubectl")
|
||||
.args([
|
||||
"--context",
|
||||
&context,
|
||||
"--namespace",
|
||||
&namespace,
|
||||
"--request-timeout=30s",
|
||||
"get",
|
||||
resource,
|
||||
])
|
||||
.args(selector)
|
||||
.args(["--output=json"])
|
||||
.kill_on_drop(true)
|
||||
.output()
|
||||
.await
|
||||
.expect("run scoped kubectl get");
|
||||
assert!(
|
||||
output.status.success(),
|
||||
"kubectl get {resource} {selector:?}: {}",
|
||||
String::from_utf8_lossy(&output.stderr)
|
||||
);
|
||||
serde_json::from_slice(&output.stdout).expect("parse kubectl JSON")
|
||||
}
|
||||
|
||||
async fn sandbox_resource(id: &str) -> Value {
|
||||
let selector = format!("openshell.ai/sandbox-id={id}");
|
||||
let list = kubectl_get("sandboxes.agents.x-k8s.io", &["--selector", &selector]).await;
|
||||
let [resource] = list["items"].as_array().expect("Sandbox items").as_slice() else {
|
||||
panic!("expected exactly one Sandbox resource for {id}: {list}");
|
||||
};
|
||||
assert_eq!(
|
||||
resource["metadata"]["labels"]["openshell.ai/sandbox-id"],
|
||||
id
|
||||
);
|
||||
assert!(
|
||||
resource["metadata"]["uid"]
|
||||
.as_str()
|
||||
.is_some_and(|uid| !uid.is_empty())
|
||||
);
|
||||
resource.clone()
|
||||
}
|
||||
|
||||
/// Read claims by their exact names from the actual workload Pod. Do not rely
|
||||
/// on controller-version-dependent PVC names or labels, or inspect other PVCs.
|
||||
async fn claim_uids(resource: &Value) -> BTreeMap<String, String> {
|
||||
let pod_name = resource["metadata"]["annotations"]["agents.x-k8s.io/pod-name"]
|
||||
.as_str()
|
||||
.unwrap_or_else(|| resource["metadata"]["name"].as_str().expect("Sandbox name"));
|
||||
let pod = kubectl_get("pod", &[pod_name]).await;
|
||||
assert!(
|
||||
pod["metadata"]["ownerReferences"]
|
||||
.as_array()
|
||||
.expect("workload Pod owners")
|
||||
.iter()
|
||||
.any(|owner| owner["uid"] == resource["metadata"]["uid"]),
|
||||
"workload Pod must belong to this Sandbox resource"
|
||||
);
|
||||
let agent = pod["spec"]["containers"]
|
||||
.as_array()
|
||||
.expect("Pod containers")
|
||||
.iter()
|
||||
.find(|container| container["name"] == "agent")
|
||||
.expect("agent container");
|
||||
let workspace_volume = agent["volumeMounts"]
|
||||
.as_array()
|
||||
.expect("agent volume mounts")
|
||||
.iter()
|
||||
.find(|mount| mount["mountPath"] == "/sandbox")
|
||||
.expect("workspace volume mount");
|
||||
let volumes = pod["spec"]["volumes"].as_array().expect("Pod volumes");
|
||||
assert!(
|
||||
volumes.iter().any(|volume| {
|
||||
volume["name"] == workspace_volume["name"]
|
||||
&& volume["persistentVolumeClaim"]["claimName"]
|
||||
.as_str()
|
||||
.is_some()
|
||||
}),
|
||||
"workspace must be backed by a PVC"
|
||||
);
|
||||
let mut claims = BTreeMap::new();
|
||||
for volume in volumes {
|
||||
let Some(name) = volume["persistentVolumeClaim"]["claimName"].as_str() else {
|
||||
continue;
|
||||
};
|
||||
let claim = kubectl_get("persistentvolumeclaim", &[name]).await;
|
||||
let uid = claim["metadata"]["uid"]
|
||||
.as_str()
|
||||
.filter(|uid| !uid.is_empty())
|
||||
.expect("PVC UID");
|
||||
assert_eq!(claim["status"]["phase"], "Bound");
|
||||
claims.insert(name.to_owned(), uid.to_owned());
|
||||
}
|
||||
assert!(
|
||||
!claims.is_empty(),
|
||||
"test must exercise PVC-backed workspace storage"
|
||||
);
|
||||
claims
|
||||
}
|
||||
|
||||
fn assert_resource_retained(before: &Value, after: &Value, image: &str) {
|
||||
assert_eq!(after["metadata"]["name"], before["metadata"]["name"]);
|
||||
assert_eq!(after["metadata"]["uid"], before["metadata"]["uid"]);
|
||||
assert_eq!(
|
||||
after["spec"]["volumeClaimTemplates"],
|
||||
before["spec"]["volumeClaimTemplates"]
|
||||
);
|
||||
let agent = after["spec"]["podTemplate"]["spec"]["containers"]
|
||||
.as_array()
|
||||
.expect("workload containers")
|
||||
.iter()
|
||||
.find(|container| container["name"] == "agent")
|
||||
.expect("agent container");
|
||||
assert_eq!(agent["image"], image);
|
||||
}
|
||||
|
||||
fn assert_identity_retained(before: &proto::Sandbox, after: &proto::Sandbox) {
|
||||
assert_eq!(
|
||||
after.metadata.as_ref().expect("metadata").id,
|
||||
before.metadata.as_ref().expect("metadata").id
|
||||
);
|
||||
assert_eq!(after.host_key_fingerprint, before.host_key_fingerprint);
|
||||
}
|
||||
|
||||
async fn assert_base_release(sandbox: &Sandbox, release: &str) {
|
||||
assert!(
|
||||
sandbox
|
||||
.exec(["cat", "/etc/alpine-release"])
|
||||
.await
|
||||
.expect("base image release")
|
||||
.stdout
|
||||
.trim()
|
||||
.starts_with(release)
|
||||
);
|
||||
}
|
||||
|
||||
fn assert_update_completed(original: &proto::Sandbox, updated: &proto::Sandbox) {
|
||||
assert_identity_retained(original, updated);
|
||||
let status = updated.status.as_ref().expect("updated status");
|
||||
assert_eq!(status.phase, proto::SandboxPhase::Stopped as i32);
|
||||
assert!(
|
||||
status
|
||||
.provisioning
|
||||
.as_ref()
|
||||
.is_none_or(|record| !record.driver_operation_pending),
|
||||
"successful image update must settle the old-understood operation interlock"
|
||||
);
|
||||
assert_eq!(
|
||||
updated
|
||||
.spec
|
||||
.as_ref()
|
||||
.expect("spec")
|
||||
.template
|
||||
.as_ref()
|
||||
.expect("template")
|
||||
.image,
|
||||
*UPDATED_IMAGE
|
||||
);
|
||||
}
|
||||
|
||||
async fn assert_replay(
|
||||
client: &mut AuthedSandboxUpdatesClient,
|
||||
update: proto::UpdateSandboxRequest,
|
||||
updated: &proto::Sandbox,
|
||||
) {
|
||||
// Replay restores current resource state, not a captured transient status.
|
||||
// Use retained driver evidence to prove that no new image operation ran.
|
||||
let id = &updated.metadata.as_ref().expect("updated metadata").id;
|
||||
let before = sandbox_resource(id).await;
|
||||
let evidence = before["metadata"]["annotations"]["openshell.ai/sandbox-image-update"].clone();
|
||||
let operation: Value =
|
||||
serde_json::from_str(evidence.as_str().expect("image operation evidence"))
|
||||
.expect("parse image operation evidence");
|
||||
assert!(
|
||||
operation["revision"]
|
||||
.as_u64()
|
||||
.is_some_and(|revision| revision > 0)
|
||||
);
|
||||
assert!(
|
||||
operation["operation_id"]
|
||||
.as_str()
|
||||
.is_some_and(|id| !id.is_empty())
|
||||
);
|
||||
let replay = client
|
||||
.update_sandbox(rpc(update))
|
||||
.await
|
||||
.expect("request-ID replay");
|
||||
assert_eq!(
|
||||
replay
|
||||
.metadata()
|
||||
.get("openshell-replayed")
|
||||
.expect("replay metadata"),
|
||||
"true"
|
||||
);
|
||||
let replayed = replay.into_inner().sandbox.expect("replayed Sandbox");
|
||||
assert_update_completed(updated, &replayed);
|
||||
assert_eq!(replayed.spec, updated.spec);
|
||||
let after = sandbox_resource(id).await;
|
||||
assert_eq!(after["metadata"]["uid"], before["metadata"]["uid"]);
|
||||
assert_eq!(
|
||||
after["metadata"]["annotations"]["openshell.ai/sandbox-image-update"], evidence,
|
||||
"replay must retain the exact driver operation identity and revision"
|
||||
);
|
||||
assert!(
|
||||
replayed
|
||||
.metadata
|
||||
.as_ref()
|
||||
.expect("replayed metadata")
|
||||
.resource_version
|
||||
>= updated
|
||||
.metadata
|
||||
.as_ref()
|
||||
.expect("updated metadata")
|
||||
.resource_version
|
||||
);
|
||||
}
|
||||
|
||||
async fn assert_image_update_supported(client: &mut AuthedGrpcClient) {
|
||||
// This test is Kubernetes-only: missing support is a regression, not a skip.
|
||||
let info = client
|
||||
.get_gateway_info(rpc(proto::GetGatewayInfoRequest {}))
|
||||
.await
|
||||
.expect("get gateway capabilities")
|
||||
.into_inner();
|
||||
assert!(info.extensions.iter().any(|extension| {
|
||||
extension
|
||||
.supported_capabilities
|
||||
.iter()
|
||||
.any(|capability| capability == "openshell.compute.update-sandbox-configuration")
|
||||
}));
|
||||
}
|
||||
|
||||
/// Updating a stopped sandbox retains storage and identity and changes the next workload.
|
||||
#[testkit::test]
|
||||
async fn image_update_retains_workspace_and_identity(cx: &mut Context) -> testkit::Result {
|
||||
let selected = testkit::Gateway::from_env()?;
|
||||
let workspace = selected.workspace.as_str();
|
||||
let gateway = selected.connect().await?;
|
||||
let mut client = gateway.raw_grpc();
|
||||
let mut image_client = gateway.raw_sandbox_updates();
|
||||
assert_image_update_supported(&mut client).await;
|
||||
|
||||
let sandbox = cx
|
||||
// The testkit's unique prefix leaves five characters of the name limit.
|
||||
.sandbox("image")
|
||||
.spec(|spec| spec.image = Some(ORIGINAL_IMAGE.to_string()))
|
||||
.create()
|
||||
.await?;
|
||||
sandbox.wait_ready().await?;
|
||||
let original = get_sandbox(&mut client, workspace, sandbox.name()).await;
|
||||
let id = &original.metadata.as_ref().expect("sandbox metadata").id;
|
||||
assert!(!id.is_empty(), "sandbox must have a canonical ID");
|
||||
assert!(
|
||||
original.host_key_fingerprint.starts_with("SHA256:"),
|
||||
"sandbox must have a gateway-managed SSH identity"
|
||||
);
|
||||
assert_base_release(&sandbox, "3.21.").await;
|
||||
let sentinel = request_id();
|
||||
sandbox
|
||||
.exec([
|
||||
"sh",
|
||||
"-c",
|
||||
&format!("printf '%s' '{sentinel}' > '{SENTINEL_PATH}'"),
|
||||
])
|
||||
.await
|
||||
.expect("write workspace sentinel");
|
||||
let before = sandbox_resource(id).await;
|
||||
let original_claims = claim_uids(&before).await;
|
||||
|
||||
let running_update = proto::UpdateSandboxRequest {
|
||||
workspace_scope: Some(proto::workspace_selector(workspace)),
|
||||
name: sandbox.name().to_owned(),
|
||||
image: Some(UPDATED_IMAGE.to_string()),
|
||||
expected_resource_version: original
|
||||
.metadata
|
||||
.as_ref()
|
||||
.expect("metadata")
|
||||
.resource_version,
|
||||
request_id: String::new(),
|
||||
..Default::default()
|
||||
};
|
||||
let error = image_client
|
||||
.update_sandbox(rpc(running_update.clone()))
|
||||
.await
|
||||
.expect_err("running update must be rejected");
|
||||
assert_eq!(error.code(), tonic::Code::FailedPrecondition);
|
||||
assert_resource_retained(&before, &sandbox_resource(id).await, &ORIGINAL_IMAGE);
|
||||
|
||||
sandbox.stop().await?;
|
||||
let stopped = get_sandbox(&mut client, workspace, sandbox.name()).await;
|
||||
let update = proto::UpdateSandboxRequest {
|
||||
expected_resource_version: stopped
|
||||
.metadata
|
||||
.as_ref()
|
||||
.expect("metadata")
|
||||
.resource_version,
|
||||
request_id: request_id(),
|
||||
command: Some(proto::SandboxCommandUpdate { argv: vec![
|
||||
"/bin/sh".into(), "-c".into(),
|
||||
"mkdir -p /sandbox/harness; printf '%s:%s' \"$AGENT_MODE\" \"$1\" > /sandbox/harness/configuration; exec sleep 86400".into(),
|
||||
"agent-bootstrap".into(), "new-argument".into(),
|
||||
] }),
|
||||
environment: Some(proto::SandboxMapUpdate { values: std::collections::HashMap::from([("AGENT_MODE".into(), "updated".into())]) }),
|
||||
labels: Some(proto::SandboxMapUpdate { values: std::collections::HashMap::from([("team".into(), "updated-agent".into())]) }),
|
||||
annotations: Some(proto::SandboxMapUpdate { values: std::collections::HashMap::from([("example.com/revision".into(), "2".into())]) }),
|
||||
..running_update
|
||||
};
|
||||
let (updated, update) =
|
||||
update_stopped_sandbox(&mut client, &mut image_client, workspace, update).await;
|
||||
assert_update_completed(&original, &updated);
|
||||
assert_resource_retained(&before, &sandbox_resource(id).await, &UPDATED_IMAGE);
|
||||
|
||||
assert_replay(&mut image_client, update, &updated).await;
|
||||
assert_recreated_resource_rejects_stale_patch(
|
||||
&sandbox_resource(id).await,
|
||||
&cx.name("uid-race"),
|
||||
)
|
||||
.await;
|
||||
|
||||
sandbox.start().await?;
|
||||
assert_base_release(&sandbox, "3.22.").await;
|
||||
assert_eq!(
|
||||
sandbox
|
||||
.exec(["cat", "/sandbox/harness/configuration"])
|
||||
.await
|
||||
.expect("updated command and environment ran on start")
|
||||
.stdout
|
||||
.trim(),
|
||||
"updated:new-argument"
|
||||
);
|
||||
let observed = get_sandbox(&mut client, workspace, sandbox.name()).await;
|
||||
assert_eq!(
|
||||
observed
|
||||
.metadata
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.labels
|
||||
.get("team")
|
||||
.map(String::as_str),
|
||||
Some("updated-agent")
|
||||
);
|
||||
assert_eq!(
|
||||
observed
|
||||
.metadata
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.annotations
|
||||
.get("example.com/revision")
|
||||
.map(String::as_str),
|
||||
Some("2")
|
||||
);
|
||||
assert_eq!(
|
||||
sandbox
|
||||
.exec(["cat", SENTINEL_PATH])
|
||||
.await
|
||||
.expect("retained workspace sentinel")
|
||||
.stdout
|
||||
.trim(),
|
||||
sentinel
|
||||
);
|
||||
let current = get_sandbox(&mut client, workspace, sandbox.name()).await;
|
||||
assert_identity_retained(&original, ¤t);
|
||||
let after = sandbox_resource(id).await;
|
||||
assert_resource_retained(&before, &after, &UPDATED_IMAGE);
|
||||
assert_eq!(
|
||||
claim_uids(&after).await,
|
||||
original_claims,
|
||||
"all mounted PVC names and UIDs must survive image replacement"
|
||||
);
|
||||
sandbox.delete().await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// A terminal command can be stopped and repaired without replacing its storage.
|
||||
#[testkit::test]
|
||||
async fn terminal_workload_can_stop_update_and_restart(cx: &mut Context) -> testkit::Result {
|
||||
let selected = testkit::Gateway::from_env()?;
|
||||
let workspace = selected.workspace.as_str();
|
||||
let gateway = selected.connect().await?;
|
||||
let mut client = gateway.raw_grpc();
|
||||
let mut updates = gateway.raw_sandbox_updates();
|
||||
let sandbox = cx
|
||||
.sandbox("term")
|
||||
.restart_policy(testkit::sdk::SandboxRestartPolicy::Never)
|
||||
.spec(|spec| spec.image = Some(ORIGINAL_IMAGE.to_string()))
|
||||
.create()
|
||||
.await?;
|
||||
sandbox.wait_ready().await?;
|
||||
let original = get_sandbox(&mut client, workspace, sandbox.name()).await;
|
||||
let before = sandbox_resource(sandbox.id()).await;
|
||||
let original_claims = claim_uids(&before).await;
|
||||
let sentinel = request_id();
|
||||
sandbox
|
||||
.exec([
|
||||
"sh",
|
||||
"-c",
|
||||
&format!("printf '%s' '{sentinel}' > '{SENTINEL_PATH}'"),
|
||||
])
|
||||
.await?;
|
||||
|
||||
for exit_code in [0, 1] {
|
||||
sandbox.stop().await?;
|
||||
update_stopped_sandbox(
|
||||
&mut client,
|
||||
&mut updates,
|
||||
workspace,
|
||||
proto::UpdateSandboxRequest {
|
||||
workspace_scope: Some(proto::workspace_selector(workspace)),
|
||||
name: sandbox.name().to_owned(),
|
||||
command: Some(proto::SandboxCommandUpdate {
|
||||
argv: vec!["/bin/sh".into(), "-c".into(), format!("exit {exit_code}")],
|
||||
}),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await;
|
||||
// Do not wait for Ready: the command may exit before Ready is observed.
|
||||
client
|
||||
.start_sandbox(rpc(proto::StartSandboxRequest {
|
||||
workspace_scope: Some(proto::workspace_selector(workspace)),
|
||||
name: sandbox.name().to_owned(),
|
||||
..Default::default()
|
||||
}))
|
||||
.await
|
||||
.expect("start terminal command");
|
||||
sandbox.wait_finished().await?;
|
||||
let terminal = get_sandbox(&mut client, workspace, sandbox.name()).await;
|
||||
let status = terminal.status.as_ref().unwrap();
|
||||
assert_eq!(
|
||||
status.phase,
|
||||
if exit_code == 0 {
|
||||
proto::SandboxPhase::Completed
|
||||
} else {
|
||||
proto::SandboxPhase::Error
|
||||
} as i32
|
||||
);
|
||||
assert_eq!(status.exit_code, Some(exit_code));
|
||||
assert!(status.conditions.iter().any(|condition| condition.reason
|
||||
== if exit_code == 0 {
|
||||
"MainProcessCompleted"
|
||||
} else {
|
||||
"MainProcessFailed"
|
||||
}));
|
||||
|
||||
sandbox.stop().await?;
|
||||
let stopped = get_sandbox(&mut client, workspace, sandbox.name()).await;
|
||||
assert_eq!(
|
||||
stopped.status.as_ref().unwrap().phase,
|
||||
proto::SandboxPhase::Stopped as i32
|
||||
);
|
||||
assert_eq!(stopped.status.as_ref().unwrap().exit_code, Some(exit_code));
|
||||
update_stopped_sandbox(
|
||||
&mut client,
|
||||
&mut updates,
|
||||
workspace,
|
||||
proto::UpdateSandboxRequest {
|
||||
workspace_scope: Some(proto::workspace_selector(workspace)),
|
||||
name: sandbox.name().to_owned(),
|
||||
command: Some(proto::SandboxCommandUpdate {
|
||||
argv: vec!["sleep".into(), "86400".into()],
|
||||
}),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await;
|
||||
sandbox.start().await?;
|
||||
let repaired = get_sandbox(&mut client, workspace, sandbox.name()).await;
|
||||
assert_identity_retained(&original, &repaired);
|
||||
assert_eq!(repaired.status.as_ref().unwrap().exit_code, None);
|
||||
assert_eq!(
|
||||
sandbox.exec(["cat", SENTINEL_PATH]).await?.stdout.trim(),
|
||||
sentinel
|
||||
);
|
||||
let after = sandbox_resource(sandbox.id()).await;
|
||||
assert_resource_retained(&before, &after, &ORIGINAL_IMAGE);
|
||||
assert_eq!(claim_uids(&after).await, original_claims);
|
||||
}
|
||||
sandbox.delete().await?;
|
||||
Ok(())
|
||||
}
|
||||
Reference in New Issue
Block a user