fix(supervisor): report why the pool could not place a sandbox (#4985)

A launch the supervisor pool could not place was refused as
AssignmentError::Capacity, so a pool with no attached member, a pool not
ready yet, and a placement failure all surfaced as "the shared supervisor
has no capacity for another sandbox instance".

Add LaunchError::Unplaced carrying the pool's reason, and report a pool with
no live member as having no attached replica instead of no replica with
room. Capacity refusals of a full host are unchanged.

Signed-off-by: divesh <dgude@nvidia.com>

crates/openshell-driver-kubernetes/src/driver.rs

@@ -3863,11 +3863,15 @@ impl KubernetesComputeDriver {
                     .await
                     .map_err(|error| match error {
                         LaunchError::Persist(error) => error,
-                        // The host is at capacity; the create fails before
-                        // its workload runs.
+                        // The host is at capacity, or the pool could not
+                        // place the instance; the create fails before its
+                        // workload runs.
                         LaunchError::Refused(error) => {
                             KubernetesDriverError::Precondition(error.to_string())
                         }
+                        LaunchError::Unplaced(reason) => {
+                            KubernetesDriverError::Precondition(reason)
+                        }
                         LaunchError::Invalid(error) => {
                             KubernetesDriverError::Message(error.to_string())
                         }

crates/openshell-server/README.md

@@ -71,9 +71,9 @@ replica, independent of which replica runs the compute driver's watch:
   the owner record, charges the member's capacity, and publishes the
   `Admitting` record, conditioned on that record's version, so an owner always
   has its published instance and a removed record never gains one. A member at
-  its 256-instance capacity is skipped. With no live member with room the
-  launch is refused as a capacity refusal: it removes the admitting record it
-  created, only at the version it created (a concurrent launch of the same
+  its 256-instance capacity is skipped. With no live member, or none with
+  room, the launch is refused with that reason: it removes the admitting
+  record it created, only at the version it created (a concurrent launch of the same
   generation that admitted it first wins), and then that record's candidate
   and material; a removal that fails is left to fencing. One
   `supervisor_instance_owner` record per instance names the member, its

crates/openshell-server/src/supervisor_pool.rs

@@ -1865,6 +1865,9 @@ impl SupervisorPool {
                 .await
                 .map_err(|error| format!("read supervisor pool capacity: {error}"))?,
         );
+        if members.is_empty() {
+            return Err("no shared supervisor replica is attached to the gateway".to_string());
+        }
         let ring = ring_of(&members);
         let targets = candidates(
             &ring,
@@ -3267,12 +3270,14 @@ mod tests {
         assert_eq!(view.members.len(), 3);
         // With no live member a launch is refused before its workload runs.
         let empty = Fixture::new().await;
+        let refused = empty
+            .replica("gw-b")
+            .place_launch(launch("rt-route-none", "sandbox-x", "generation-1"))
+            .await
+            .unwrap_err();
         assert!(
-            empty
-                .replica("gw-b")
-                .place_launch(launch("rt-route-none", "sandbox-x", "generation-1"))
-                .await
-                .is_err()
+            refused.contains("no shared supervisor replica is attached"),
+            "{refused}"
         );
     }
 

crates/openshell-supervisor-control/README.md

@@ -66,8 +66,8 @@ re-implementing the control contracts.
   only then assigns the instance; a host at capacity is refused before the
   workload runs. For a pool, the gateway's installed `InstancePlacement`
   durably records the instance and its launch term's owner before the
-  assignment, and a launch no member can take is refused like a full host
-  (`LaunchError::Refused`). A pool's withdrawals are removed from the durable
+  assignment, and a launch no member can take fails with the pool's reason
+  (`LaunchError::Unplaced`). A pool's withdrawals are removed from the durable
   records by the gateway, and a recovery reconciles those records with the
   driver's listing instead of replacing the desired set. `begin_recovery` captures
   the revision before the driver lists its durable records and

crates/openshell-supervisor-control/src/coordinator.rs

@@ -48,11 +48,16 @@ pub enum LaunchError<E> {
     /// The driver could not persist the launch material; nothing was assigned.
     #[error("{0}")]
     Persist(E),
-    /// The host, or every member of a pool, is at capacity or unavailable.
+    /// The host is at capacity. The material was persisted; the caller rolls
+    /// the generation back before its workload runs.
+    #[error("{0}")]
+    Refused(AssignmentError),
+    /// The pool could not place the instance: it is not ready, no member is
+    /// attached or has room, or its records could not be read or written.
     /// The material was persisted; the caller rolls the generation back
     /// before its workload runs.
     #[error("{0}")]
-    Refused(AssignmentError),
+    Unplaced(String),
 }
 
 /// A new pool instance to record durably before it is assigned.
@@ -196,10 +201,10 @@ impl SharedSupervision {
     /// assign the instance so the host starts it. The launch material is
     /// persisted first so a restarted gateway recovers the assignment from the
     /// generation's own records. The pool records the instance durably for
-    /// every gateway replica, on a live member with room; without one, or
-    /// beyond the pool's capacity, the launch is refused as
-    /// [`LaunchError::Refused`], so the caller fails its create before the
-    /// workload runs. An identical assignment changes nothing.
+    /// every gateway replica, on a live member with room; without one the
+    /// launch fails as [`LaunchError::Unplaced`], and beyond the pool's
+    /// capacity as [`LaunchError::Refused`], so the caller fails its create
+    /// before the workload runs. An identical assignment changes nothing.
     pub async fn launch<E, F>(
         &self,
         registration: &SandboxInstanceRegistration,
@@ -218,11 +223,9 @@ impl SharedSupervision {
         // any member is assigned it, so every gateway replica serves it and a
         // later failure of that member is recovered by takeover.
         let Some(placement) = self.controller.placement() else {
-            tracing::warn!(
-                route = %registration.route,
-                "the supervisor pool is not ready to place sandbox instances"
-            );
-            return Err(LaunchError::Refused(AssignmentError::Capacity));
+            const NOT_READY: &str = "the supervisor pool is not ready to place sandbox instances";
+            tracing::warn!(route = %registration.route, "{NOT_READY}");
+            return Err(LaunchError::Unplaced(NOT_READY.to_string()));
         };
         if let Err(error) = placement
             .place_launch(PlacementRequest {
@@ -237,7 +240,7 @@ impl SharedSupervision {
                 %error,
                 "could not place a shared sandbox instance in the supervisor pool"
             );
-            return Err(LaunchError::Refused(AssignmentError::Capacity));
+            return Err(LaunchError::Unplaced(error));
         }
         let route = assignment.route.clone();
         self.controller.assign(assignment).map_err(|error| {