diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index 41b7a6603..57be9ec72 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -286,13 +286,16 @@ should read it for binary-scoped policy decisions; if allowed network rules are all denied, inspect that file and the network sidecar logs. If `supervisor_topology = "split-pod"` is rendered, each sandbox should have a -separate supervisor pod, a headless supervisor Service, a proxy CA Secret, and -two per-sandbox NetworkPolicies. The agent pod should have -`openshell.ai/sandbox-role=agent`; the supervisor pod should have +separate supervisor Deployment with one supervisor pod, a headless supervisor +Service, a proxy CA Secret, and two per-sandbox NetworkPolicies. The agent pod +should have `openshell.ai/sandbox-role=agent`; the supervisor pod should have `openshell.ai/sandbox-role=supervisor`; both should share the same -`openshell.ai/sandbox-id`. The supervisor pod must have a controlling -`Sandbox` ownerReference and the `openshell.io/sandbox-id` annotation so the -TokenReview bootstrap path can mint a sandbox JWT. If the agent cannot reach +`openshell.ai/sandbox-id`. The supervisor Deployment must have a controlling +`Sandbox` ownerReference. The Deployment pod template must carry the +`openshell.io/sandbox-id` annotation so the TokenReview bootstrap path can mint +a sandbox JWT. For supervisor pods, the gateway validates the +`Pod -> ReplicaSet -> Deployment -> Sandbox` owner chain, so missing +`apps/replicasets get` RBAC can also break bootstrap. If the agent cannot reach the gateway, check DNS to the headless Service, the agent egress NetworkPolicy DNS exception for kube-dns/CoreDNS, and the supervisor ingress NetworkPolicy allowing only that agent pod on ports `3128` and `18080`. diff --git a/.agents/skills/helm-dev-environment/SKILL.md b/.agents/skills/helm-dev-environment/SKILL.md index 2429922b3..06473e02b 100644 --- a/.agents/skills/helm-dev-environment/SKILL.md +++ b/.agents/skills/helm-dev-environment/SKILL.md @@ -74,10 +74,11 @@ All Skaffold commands build the `gateway` and `supervisor` images and deploy the chart. The sidecar profile renders an `openshell-network-init` init container for nftables setup and a non-root `openshell-supervisor-network` runtime sidecar for proxying. The split-pod profile renders network supervision in a separate -supervisor pod and relies on Kubernetes NetworkPolicy enforcement so the agent -pod can reach only its paired supervisor plus DNS. The default local k3s/k3d -cluster keeps k3s's embedded NetworkPolicy controller enabled; if you replace -the CNI, install a policy-enforcing CNI before using split-pod. The +supervisor Deployment with one pod and relies on Kubernetes NetworkPolicy +enforcement so the agent pod can reach only its paired supervisor plus DNS. The +default local k3s/k3d cluster keeps k3s's embedded NetworkPolicy controller +enabled; if you replace the CNI, install a policy-enforcing CNI before using +split-pod. The `pkiInitJob` hook (a pre-install Job that runs `openshell-gateway generate-certs`) generates mTLS secrets on first install. Envoy Gateway opt-in; see the Optional Add-ons section below. diff --git a/architecture/gateway.md b/architecture/gateway.md index d873b2a10..6927d41ff 100644 --- a/architecture/gateway.md +++ b/architecture/gateway.md @@ -64,9 +64,11 @@ Podman, and VM drivers deliver the initial token through supervisor-only runtime material; Kubernetes supervisors exchange a projected ServiceAccount token through `IssueSandboxToken`. The gateway validates that projected token with Kubernetes `TokenReview`, requires the configured sandbox service account, -checks the returned pod binding against the live pod UID, and verifies the pod's -controlling `Sandbox` ownerReference against the live Sandbox CR UID and -sandbox-id label before minting the gateway JWT. The bootstrap path accepts +checks the returned pod binding against the live pod UID, and verifies the +pod's ownership against the live Sandbox CR UID and sandbox-id label before +minting the gateway JWT. Agent pods must be directly controlled by the +`Sandbox` CR. Split-pod supervisor pods may be controlled through the Kubernetes +`Pod -> ReplicaSet -> Deployment -> Sandbox` chain. The bootstrap path accepts both `agents.x-k8s.io/v1beta1` ownerReferences from newer Agent Sandbox controllers and `agents.x-k8s.io/v1alpha1` ownerReferences from existing deployments. Supervisors renew gateway JWTs in memory before expiry only while diff --git a/crates/openshell-driver-kubernetes/README.md b/crates/openshell-driver-kubernetes/README.md index 21410b4bb..aa3a9338d 100644 --- a/crates/openshell-driver-kubernetes/README.md +++ b/crates/openshell-driver-kubernetes/README.md @@ -67,11 +67,12 @@ filesystem policy, process privilege dropping, or process/binary identity checks. Network endpoint and L7 policy remain enforced by the network sidecar. The `split-pod` supervisor topology runs network enforcement and gateway -forwarding in a separate supervisor pod. The agent pod runs only the -process-mode supervisor and reaches the supervisor through a per-sandbox -headless Service. The driver creates owner-referenced supervisor Pod, Service, -proxy CA Secret, and NetworkPolicy resources so agent egress is limited to its -paired supervisor pod plus DNS. +forwarding in a separate supervisor Deployment with one pod. The agent pod runs +only the process-mode supervisor and reaches the supervisor through a +per-sandbox headless Service. The driver creates an owner-referenced supervisor +Deployment with one replica plus Service, proxy CA Secret, and NetworkPolicy +resources so agent egress is limited to its paired supervisor pod plus DNS. If +the supervisor pod is deleted, the Deployment recreates it. Sidecar mode uses the pod `fsGroup` to make the projected service-account token and sandbox client TLS secret group-readable so the non-root process supervisor diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index 311145b71..d6e931c0a 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -10,7 +10,8 @@ use crate::config::{ SupervisorTopology, }; use futures::{Stream, StreamExt, TryStreamExt}; -use k8s_openapi::api::core::v1::{Event as KubeEventObj, Namespace, Node, Pod, Secret, Service}; +use k8s_openapi::api::apps::v1::Deployment; +use k8s_openapi::api::core::v1::{Event as KubeEventObj, Namespace, Node, Secret, Service}; use k8s_openapi::api::networking::v1::NetworkPolicy; use kube::api::{Api, ApiResource, DeleteParams, ListParams, PostParams}; use kube::core::gvk::GroupVersionKind; @@ -649,7 +650,7 @@ impl KubernetesComputeDriver { .map(|template| template.environment.clone()) .unwrap_or_default(); let spec_environment = spec_pod_env(spec); - let pod_owner_ref = split_owner_reference(sandbox_cr, sandbox_api_version, true)?; + let deployment_owner_ref = split_owner_reference(sandbox_cr, sandbox_api_version, true)?; let dependent_owner_ref = split_owner_reference(sandbox_cr, sandbox_api_version, false)?; let (ca_cert_pem, ca_key_pem) = generate_split_proxy_ca()?; @@ -665,19 +666,20 @@ impl KubernetesComputeDriver { split_agent_egress_network_policy(&names, params, dependent_owner_ref.clone()); let supervisor_ingress = split_supervisor_ingress_network_policy(&names, params, dependent_owner_ref); - let supervisor_pod = split_supervisor_pod( + let supervisor_deployment = split_supervisor_deployment( &names, &template_environment, &spec_environment, params, - pod_owner_ref, + deployment_owner_ref, ); let secrets: Api = Api::namespaced(self.client.clone(), &self.config.namespace); let services: Api = Api::namespaced(self.client.clone(), &self.config.namespace); let policies: Api = Api::namespaced(self.client.clone(), &self.config.namespace); - let pods: Api = Api::namespaced(self.client.clone(), &self.config.namespace); + let deployments: Api = + Api::namespaced(self.client.clone(), &self.config.namespace); tokio::time::timeout( KUBE_API_TIMEOUT, @@ -729,12 +731,12 @@ impl KubernetesComputeDriver { .map_err(KubernetesDriverError::from_kube)?; tokio::time::timeout( KUBE_API_TIMEOUT, - pods.create(&PostParams::default(), &supervisor_pod), + deployments.create(&PostParams::default(), &supervisor_deployment), ) .await .map_err(|_| { KubernetesDriverError::Message(format!( - "timed out after {}s creating split-pod supervisor pod", + "timed out after {}s creating split-pod supervisor deployment", KUBE_API_TIMEOUT.as_secs() )) })? @@ -743,7 +745,7 @@ impl KubernetesComputeDriver { info!( sandbox_id = %sandbox.id, sandbox_name = %sandbox.name, - supervisor_pod = %names.supervisor_pod, + supervisor_deployment = %names.supervisor_deployment, service = %names.service, "Created split-pod supervisor resources" ); @@ -756,11 +758,12 @@ impl KubernetesComputeDriver { let services: Api = Api::namespaced(self.client.clone(), &self.config.namespace); let policies: Api = Api::namespaced(self.client.clone(), &self.config.namespace); - let pods: Api = Api::namespaced(self.client.clone(), &self.config.namespace); + let deployments: Api = + Api::namespaced(self.client.clone(), &self.config.namespace); let _ = tokio::time::timeout( KUBE_API_TIMEOUT, - pods.delete(&names.supervisor_pod, &DeleteParams::default()), + deployments.delete(&names.supervisor_deployment, &DeleteParams::default()), ) .await; let _ = tokio::time::timeout( @@ -1460,7 +1463,7 @@ fn gateway_tls_server_name(grpc_endpoint: &str) -> Option { #[derive(Debug, Clone)] struct SplitPodResourceNames { - supervisor_pod: String, + supervisor_deployment: String, service: String, proxy_ca_secret: String, agent_egress_network_policy: String, @@ -1469,7 +1472,7 @@ struct SplitPodResourceNames { fn split_pod_resource_names(sandbox_name: &str) -> SplitPodResourceNames { SplitPodResourceNames { - supervisor_pod: dns_label_name("os-sup", sandbox_name), + supervisor_deployment: dns_label_name("os-sup", sandbox_name), service: dns_label_name("os-svc", sandbox_name), proxy_ca_secret: dns_label_name("os-ca", sandbox_name), agent_egress_network_policy: dns_label_name("os-eg", sandbox_name), @@ -3158,13 +3161,13 @@ fn split_supervisor_service( })) } -fn split_supervisor_pod( +fn split_supervisor_deployment( names: &SplitPodResourceNames, template_environment: &std::collections::HashMap, spec_environment: &std::collections::HashMap, params: &SandboxPodParams<'_>, owner_ref: serde_json::Value, -) -> Pod { +) -> Deployment { let mut container = serde_json::json!({ "name": SUPERVISOR_NETWORK_SIDECAR_NAME, "image": params.supervisor_image, @@ -3302,16 +3305,30 @@ fn split_supervisor_pod( } k8s_object(serde_json::json!({ - "apiVersion": "v1", - "kind": "Pod", + "apiVersion": "apps/v1", + "kind": "Deployment", "metadata": split_pod_object_meta( - &names.supervisor_pod, + &names.supervisor_deployment, params.namespace, params.sandbox_id, SANDBOX_ROLE_SUPERVISOR, owner_ref ), - "spec": spec + "spec": { + "replicas": 1, + "selector": { + "matchLabels": split_match_labels(params.sandbox_id, SANDBOX_ROLE_SUPERVISOR) + }, + "template": { + "metadata": { + "labels": split_labels(params.sandbox_id, SANDBOX_ROLE_SUPERVISOR), + "annotations": { + "openshell.io/sandbox-id": params.sandbox_id + } + }, + "spec": spec + } + } })) } @@ -4512,7 +4529,7 @@ mod tests { "blockOwnerDeletion": false }); - let supervisor = serde_json::to_value(split_supervisor_pod( + let supervisor = serde_json::to_value(split_supervisor_deployment( &names, &std::collections::HashMap::new(), &std::collections::HashMap::new(), @@ -4532,15 +4549,25 @@ mod tests { supervisor["metadata"]["labels"][LABEL_SANDBOX_ROLE], SANDBOX_ROLE_SUPERVISOR ); + assert_eq!(supervisor["kind"], "Deployment"); + assert_eq!(supervisor["spec"]["replicas"], 1); assert_eq!( - supervisor["spec"]["hostAliases"][0]["ip"], + supervisor["spec"]["selector"]["matchLabels"][LABEL_SANDBOX_ROLE], + SANDBOX_ROLE_SUPERVISOR + ); + assert_eq!( + supervisor["spec"]["template"]["metadata"]["labels"][LABEL_SANDBOX_ROLE], + SANDBOX_ROLE_SUPERVISOR + ); + assert_eq!( + supervisor["spec"]["template"]["spec"]["hostAliases"][0]["ip"], params.host_gateway_ip ); - let hostnames = supervisor["spec"]["hostAliases"][0]["hostnames"] + let hostnames = supervisor["spec"]["template"]["spec"]["hostAliases"][0]["hostnames"] .as_array() .unwrap(); assert!(hostnames.contains(&serde_json::json!("host.openshell.internal"))); - let container = &supervisor["spec"]["containers"][0]; + let container = &supervisor["spec"]["template"]["spec"]["containers"][0]; assert_eq!( rendered_env(container, openshell_core::sandbox_env::PROXY_BIND_ADDR), Some("0.0.0.0:3128") diff --git a/crates/openshell-server/src/auth/k8s_sa.rs b/crates/openshell-server/src/auth/k8s_sa.rs index eed0e5f08..f27b90067 100644 --- a/crates/openshell-server/src/auth/k8s_sa.rs +++ b/crates/openshell-server/src/auth/k8s_sa.rs @@ -5,8 +5,9 @@ //! //! Path-scoped to `IssueSandboxToken`. Validates a projected SA token //! presented by a sandbox pod, reads the pod's `openshell.io/sandbox-id` -//! annotation, verifies the pod is controlled by the corresponding Sandbox CR, -//! and returns a [`Principal::Sandbox`] with +//! annotation, verifies the pod is controlled by the corresponding Sandbox CR +//! either directly or through a supervisor Deployment controller chain, and +//! returns a [`Principal::Sandbox`] with //! [`SandboxIdentitySource::K8sServiceAccount`]. The `IssueSandboxToken` handler //! then mints a gateway-signed JWT for that sandbox id; subsequent gRPC calls //! from the supervisor use the gateway-minted JWT validated by @@ -19,10 +20,11 @@ use super::authenticator::Authenticator; use super::principal::{Principal, SandboxIdentitySource, SandboxPrincipal}; use async_trait::async_trait; use k8s_openapi::api::{ + apps::v1::{Deployment, ReplicaSet}, authentication::v1::{TokenReview, TokenReviewSpec, TokenReviewStatus, UserInfo}, core::v1::Pod, }; -use k8s_openapi::apimachinery::pkg::apis::meta::v1::ObjectMeta; +use k8s_openapi::apimachinery::pkg::apis::meta::v1::{ObjectMeta, OwnerReference}; use kube::Error as KubeError; use kube::api::{Api, ApiResource, PostParams}; use kube::core::{DynamicObject, gvk::GroupVersionKind}; @@ -45,7 +47,10 @@ const SANDBOX_API_VERSION_V1BETA1: &str = "v1beta1"; const SANDBOX_API_VERSION_V1ALPHA1: &str = "v1alpha1"; const SANDBOX_API_VERSION_FULL_V1BETA1: &str = "agents.x-k8s.io/v1beta1"; const SANDBOX_API_VERSION_FULL_V1ALPHA1: &str = "agents.x-k8s.io/v1alpha1"; +const APPS_API_VERSION_FULL_V1: &str = "apps/v1"; const SANDBOX_KIND: &str = "Sandbox"; +const REPLICA_SET_KIND: &str = "ReplicaSet"; +const DEPLOYMENT_KIND: &str = "Deployment"; const SANDBOX_ID_LABEL: &str = "openshell.ai/sandbox-id"; const POD_NAME_EXTRA: &str = "authentication.kubernetes.io/pod-name"; const POD_UID_EXTRA: &str = "authentication.kubernetes.io/pod-uid"; @@ -148,11 +153,21 @@ struct SandboxOwnerReference { uid: String, } +#[derive(Debug, Clone, PartialEq, Eq)] +struct ControllerOwnerReference { + api_version: String, + kind: String, + name: String, + uid: String, +} + /// Resolver backed by the apiserver's `TokenReview` API and `kube::Client` /// for the per-pod annotation lookup. pub struct LiveK8sResolver { token_reviews_api: Api, pods_api: Api, + replica_sets_api: Api, + deployments_api: Api, sandboxes_api_v1beta1: Api, sandboxes_api_v1alpha1: Api, expected_audience: String, @@ -169,6 +184,8 @@ impl LiveK8sResolver { ) -> Self { let token_reviews_api: Api = Api::all(client.clone()); let pods_api: Api = Api::namespaced(client.clone(), namespace); + let replica_sets_api: Api = Api::namespaced(client.clone(), namespace); + let deployments_api: Api = Api::namespaced(client.clone(), namespace); let sandbox_gvk_v1beta1 = GroupVersionKind::gvk(SANDBOX_API_GROUP, SANDBOX_API_VERSION_V1BETA1, SANDBOX_KIND); let sandbox_resource_v1beta1 = ApiResource::from_gvk(&sandbox_gvk_v1beta1); @@ -185,6 +202,8 @@ impl LiveK8sResolver { Self { token_reviews_api, pods_api, + replica_sets_api, + deployments_api, sandboxes_api_v1beta1, sandboxes_api_v1alpha1, expected_audience, @@ -214,6 +233,129 @@ impl LiveK8sResolver { Ok(None) } + + async fn sandbox_owner_for_pod( + &self, + pod: &Pod, + pod_name: &str, + ) -> Result { + match direct_sandbox_owner_reference(pod) { + Ok(owner) => Ok(owner), + Err(err) => { + let Some(controller) = controller_owner_reference( + pod.metadata.owner_references.as_deref().unwrap_or_default(), + ) else { + return Err(err); + }; + if controller.api_version != APPS_API_VERSION_FULL_V1 + || controller.kind != REPLICA_SET_KIND + { + return Err(err); + } + self.sandbox_owner_for_replica_set_controller(&controller, pod_name) + .await + } + } + } + + async fn sandbox_owner_for_replica_set_controller( + &self, + replica_set_owner: &ControllerOwnerReference, + pod_name: &str, + ) -> Result { + let replica_set = self + .replica_sets_api + .get_opt(&replica_set_owner.name) + .await + .map_err(|e| { + warn!( + pod = %pod_name, + replica_set = %replica_set_owner.name, + error = %e, + "failed to fetch ReplicaSet for pod identity validation" + ); + Status::internal(format!("replicaset GET failed: {e}")) + })? + .ok_or_else(|| { + warn!( + pod = %pod_name, + replica_set = %replica_set_owner.name, + "pod controller ReplicaSet was not found" + ); + Status::permission_denied("pod controller ReplicaSet not found") + })?; + validate_object_uid( + replica_set.metadata.uid.as_deref().unwrap_or_default(), + &replica_set_owner.uid, + "pod controller ReplicaSet UID mismatch", + )?; + + let deployment_owner = controller_owner_reference( + replica_set + .metadata + .owner_references + .as_deref() + .unwrap_or_default(), + ) + .ok_or_else(|| { + warn!( + pod = %pod_name, + replica_set = %replica_set_owner.name, + "ReplicaSet has no controlling Deployment ownerReference" + ); + Status::permission_denied("ReplicaSet is not controlled by a Deployment") + })?; + if deployment_owner.api_version != APPS_API_VERSION_FULL_V1 + || deployment_owner.kind != DEPLOYMENT_KIND + { + warn!( + pod = %pod_name, + replica_set = %replica_set_owner.name, + owner_api_version = %deployment_owner.api_version, + owner_kind = %deployment_owner.kind, + "ReplicaSet controller is not an apps/v1 Deployment" + ); + return Err(Status::permission_denied( + "ReplicaSet is not controlled by a Deployment", + )); + } + + let deployment = self + .deployments_api + .get_opt(&deployment_owner.name) + .await + .map_err(|e| { + warn!( + pod = %pod_name, + deployment = %deployment_owner.name, + error = %e, + "failed to fetch Deployment for pod identity validation" + ); + Status::internal(format!("deployment GET failed: {e}")) + })? + .ok_or_else(|| { + warn!( + pod = %pod_name, + deployment = %deployment_owner.name, + "ReplicaSet controller Deployment was not found" + ); + Status::permission_denied("ReplicaSet controller Deployment not found") + })?; + validate_object_uid( + deployment.metadata.uid.as_deref().unwrap_or_default(), + &deployment_owner.uid, + "ReplicaSet controller Deployment UID mismatch", + )?; + + sandbox_owner_reference_from_owner_refs( + deployment + .metadata + .owner_references + .as_deref() + .unwrap_or_default(), + "Deployment", + ) + } } #[async_trait] @@ -293,7 +435,7 @@ impl K8sIdentityResolver for LiveK8sResolver { let sandbox_id = pod_sandbox_id(&pod)?; - let owner = sandbox_owner_reference(&pod)?; + let owner = self.sandbox_owner_for_pod(&pod, &identity.pod_name).await?; let sandbox_cr = self.get_sandbox_cr_for_owner(&owner).await.map_err(|e| { warn!( pod = %identity.pod_name, @@ -406,8 +548,18 @@ fn pod_sandbox_id(pod: &Pod) -> Result { } #[allow(clippy::result_large_err)] -fn sandbox_owner_reference(pod: &Pod) -> Result { - let owner_refs = pod.metadata.owner_references.as_deref().unwrap_or_default(); +fn direct_sandbox_owner_reference(pod: &Pod) -> Result { + sandbox_owner_reference_from_owner_refs( + pod.metadata.owner_references.as_deref().unwrap_or_default(), + "pod", + ) +} + +#[allow(clippy::result_large_err)] +fn sandbox_owner_reference_from_owner_refs( + owner_refs: &[OwnerReference], + object_kind: &str, +) -> Result { let mut sandbox_refs = owner_refs .iter() .filter(|owner| is_supported_sandbox_owner_reference(owner)); @@ -424,27 +576,28 @@ fn sandbox_owner_reference(pod: &Pod) -> Result { SANDBOX_API_VERSION_FULL_V1BETA1, SANDBOX_API_VERSION_FULL_V1ALPHA1, ], - "pod Sandbox ownerReference uses unsupported apiVersion" + object_kind = %object_kind, + "Sandbox ownerReference uses unsupported apiVersion" ); } - return Err(Status::permission_denied( - "pod is not controlled by an OpenShell Sandbox", - )); + return Err(Status::permission_denied(format!( + "{object_kind} is not controlled by an OpenShell Sandbox" + ))); }; if sandbox_refs.next().is_some() { - return Err(Status::permission_denied( - "pod has multiple OpenShell Sandbox owners", - )); + return Err(Status::permission_denied(format!( + "{object_kind} has multiple OpenShell Sandbox owners" + ))); } if owner.controller != Some(true) { - return Err(Status::permission_denied( - "pod Sandbox ownerReference is not controlling", - )); + return Err(Status::permission_denied(format!( + "{object_kind} Sandbox ownerReference is not controlling" + ))); } if owner.name.is_empty() || owner.uid.is_empty() { - return Err(Status::permission_denied( - "pod Sandbox ownerReference is incomplete", - )); + return Err(Status::permission_denied(format!( + "{object_kind} Sandbox ownerReference is incomplete" + ))); } Ok(SandboxOwnerReference { api_version: owner.api_version.clone(), @@ -453,9 +606,32 @@ fn sandbox_owner_reference(pod: &Pod) -> Result { }) } -fn is_supported_sandbox_owner_reference( - owner: &k8s_openapi::apimachinery::pkg::apis::meta::v1::OwnerReference, -) -> bool { +fn controller_owner_reference(owner_refs: &[OwnerReference]) -> Option { + let owner = owner_refs + .iter() + .find(|owner| owner.controller == Some(true))?; + Some(ControllerOwnerReference { + api_version: owner.api_version.clone(), + kind: owner.kind.clone(), + name: owner.name.clone(), + uid: owner.uid.clone(), + }) +} + +#[allow(clippy::result_large_err)] +fn validate_object_uid(actual_uid: &str, expected_uid: &str, message: &str) -> Result<(), Status> { + if actual_uid != expected_uid { + warn!( + expected_uid = %expected_uid, + actual_uid = %actual_uid, + %message + ); + return Err(Status::permission_denied(message.to_string())); + } + Ok(()) +} + +fn is_supported_sandbox_owner_reference(owner: &OwnerReference) -> bool { owner.kind == SANDBOX_KIND && matches!( owner.api_version.as_str(), @@ -629,6 +805,17 @@ mod tests { } } + fn app_controller_owner(kind: &str, name: &str, uid: &str) -> OwnerReference { + OwnerReference { + api_version: APPS_API_VERSION_FULL_V1.to_string(), + block_owner_deletion: None, + controller: Some(true), + kind: kind.to_string(), + name: name.to_string(), + uid: uid.to_string(), + } + } + fn pod_with_owner_refs(owner_references: Vec) -> Pod { Pod { metadata: ObjectMeta { @@ -780,7 +967,7 @@ mod tests { fn sandbox_owner_reference_extracts_controlling_sandbox_owner() { let pod = pod_with_owner_refs(vec![sandbox_owner("sandbox-a", "cr-uid-a")]); - let owner = sandbox_owner_reference(&pod).expect("expected Sandbox owner"); + let owner = direct_sandbox_owner_reference(&pod).expect("expected Sandbox owner"); assert_eq!( owner, @@ -800,7 +987,7 @@ mod tests { "cr-uid-a", )]); - let owner = sandbox_owner_reference(&pod).expect("expected v1alpha1 Sandbox owner"); + let owner = direct_sandbox_owner_reference(&pod).expect("expected v1alpha1 Sandbox owner"); assert_eq!( owner, @@ -816,7 +1003,7 @@ mod tests { fn sandbox_owner_reference_rejects_missing_owner() { let pod = pod_with_owner_refs(vec![]); - let err = sandbox_owner_reference(&pod).expect_err("missing owner must fail"); + let err = direct_sandbox_owner_reference(&pod).expect_err("missing owner must fail"); assert_eq!(err.code(), tonic::Code::PermissionDenied); } @@ -829,8 +1016,8 @@ mod tests { "cr-uid-a", )]); - let err = - sandbox_owner_reference(&pod).expect_err("unsupported apiVersion must fail closed"); + let err = direct_sandbox_owner_reference(&pod) + .expect_err("unsupported apiVersion must fail closed"); assert_eq!(err.code(), tonic::Code::PermissionDenied); } @@ -841,7 +1028,7 @@ mod tests { owner.controller = Some(false); let pod = pod_with_owner_refs(vec![owner]); - let err = sandbox_owner_reference(&pod).expect_err("non-controller owner must fail"); + let err = direct_sandbox_owner_reference(&pod).expect_err("non-controller owner must fail"); assert_eq!(err.code(), tonic::Code::PermissionDenied); } @@ -853,11 +1040,50 @@ mod tests { sandbox_owner("sandbox-b", "cr-uid-b"), ]); - let err = sandbox_owner_reference(&pod).expect_err("multiple owners must fail"); + let err = direct_sandbox_owner_reference(&pod).expect_err("multiple owners must fail"); assert_eq!(err.code(), tonic::Code::PermissionDenied); } + #[test] + fn controller_owner_reference_extracts_controlling_apps_owner() { + let pod = pod_with_owner_refs(vec![app_controller_owner( + REPLICA_SET_KIND, + "supervisor-rs", + "rs-uid", + )]); + + let owner = controller_owner_reference(pod.metadata.owner_references.as_deref().unwrap()) + .expect("expected controller owner"); + + assert_eq!( + owner, + ControllerOwnerReference { + api_version: APPS_API_VERSION_FULL_V1.to_string(), + kind: REPLICA_SET_KIND.to_string(), + name: "supervisor-rs".to_string(), + uid: "rs-uid".to_string(), + } + ); + } + + #[test] + fn sandbox_owner_reference_from_deployment_requires_controlling_sandbox_owner() { + let deployment_owner_refs = vec![sandbox_owner("sandbox-a", "cr-uid-a")]; + + let owner = sandbox_owner_reference_from_owner_refs(&deployment_owner_refs, "Deployment") + .expect("expected Deployment Sandbox owner"); + + assert_eq!( + owner, + SandboxOwnerReference { + api_version: SANDBOX_API_VERSION_FULL_V1BETA1.to_string(), + name: "sandbox-a".to_string(), + uid: "cr-uid-a".to_string(), + } + ); + } + #[test] fn validate_sandbox_owner_reference_requires_matching_cr_uid_and_label() { let owner = SandboxOwnerReference { diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index fc4947acb..a70e2b8d8 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -238,7 +238,7 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | supervisor.image.tag | string | `""` | Supervisor image tag. Defaults to the chart appVersion when empty. | | supervisor.sidecarProxyUid | int | `1337` | UID for the long-running network sidecar/split supervisor. In sidecar topology, the network init container installs nftables rules that exempt this UID. | | supervisor.sideloadMethod | string | `""` | How the supervisor binary is delivered into sandbox pods. Empty (default) = auto-detect from cluster version: K8s >= v1.35 -> "image-volume" (ImageVolume enabled by default; GA in v1.36) K8s < v1.35 -> "init-container" (copies via init container + emptyDir) On K8s v1.33-v1.34 with the ImageVolume feature gate manually enabled, set this to "image-volume" explicitly. | -| supervisor.topology | string | `"combined"` | Supervisor pod topology for Kubernetes sandboxes. "combined" runs the current single supervisor container in the agent pod. "sidecar" runs network enforcement in a dedicated sidecar and the process supervisor as a low-capability wrapper in the agent container. "split-pod" runs network enforcement in a separate supervisor pod and restricts the agent pod to that supervisor through NetworkPolicy. | +| supervisor.topology | string | `"combined"` | Supervisor pod topology for Kubernetes sandboxes. "combined" runs the current single supervisor container in the agent pod. "sidecar" runs network enforcement in a dedicated sidecar and the process supervisor as a low-capability wrapper in the agent container. "split-pod" runs network enforcement in a separate supervisor Deployment and restricts the agent pod to that supervisor through NetworkPolicy. | | tolerations | list | `[]` | Tolerations for the gateway pod. | | workload.allowMultiReplicaStatefulSet | bool | `false` | Allow replicaCount > 1 while rendering a StatefulSet. Prefer workload.kind=deployment for external database-backed multi-replica gateways; this override exists for operators who explicitly require StatefulSet identity or storage semantics. | | workload.kind | string | `"statefulset"` | Gateway workload controller kind. Use `statefulset` for the default SQLite database, or `deployment` when server.externalDbSecret points at an external database. | diff --git a/deploy/helm/openshell/templates/role.yaml b/deploy/helm/openshell/templates/role.yaml index 85bcb1cef..80050da35 100644 --- a/deploy/helm/openshell/templates/role.yaml +++ b/deploy/helm/openshell/templates/role.yaml @@ -35,19 +35,36 @@ rules: # returned pod name and UID to the pod's `openshell.io/sandbox-id` # annotation. patch is intentionally NOT granted — the annotation is set # once at pod create and must remain immutable for the lifetime of the - # sandbox. + # sandbox. create/delete/list/watch are intentionally not granted; the Agent + # Sandbox controller creates agent pods, and split-pod supervisors are + # managed through per-sandbox Deployments. - apiGroups: - "" resources: - pods + verbs: + - get + # Split-pod topology creates one supervisor Deployment, one supervisor + # Service, and one CA Secret per sandbox. All are owner-referenced to the + # Sandbox CR for garbage collection. The gateway also reads the generated + # ReplicaSet during K8s ServiceAccount bootstrap to verify the supervisor + # pod's Pod -> ReplicaSet -> Deployment -> Sandbox owner chain. + - apiGroups: + - apps + resources: + - deployments verbs: - create - delete - get - list - watch - # Split-pod topology creates one supervisor Service and one CA Secret per - # sandbox. Both are owner-referenced to the Sandbox CR for garbage collection. + - apiGroups: + - apps + resources: + - replicasets + verbs: + - get - apiGroups: - "" resources: diff --git a/deploy/helm/openshell/tests/sandbox_namespace_test.yaml b/deploy/helm/openshell/tests/sandbox_namespace_test.yaml index ee89fce53..5cc0966b8 100644 --- a/deploy/helm/openshell/tests/sandbox_namespace_test.yaml +++ b/deploy/helm/openshell/tests/sandbox_namespace_test.yaml @@ -57,6 +57,49 @@ tests: path: metadata.namespace value: other-ns + - it: grants only pod get for sandbox token bootstrap + template: templates/role.yaml + asserts: + - contains: + path: rules + content: + apiGroups: + - "" + resources: + - pods + verbs: + - get + + - it: grants sandbox RBAC for split-pod supervisor Deployments + template: templates/role.yaml + asserts: + - contains: + path: rules + content: + apiGroups: + - apps + resources: + - deployments + verbs: + - create + - delete + - get + - list + - watch + + - it: grants ReplicaSet get for split-pod supervisor token bootstrap + template: templates/role.yaml + asserts: + - contains: + path: rules + content: + apiGroups: + - apps + resources: + - replicasets + verbs: + - get + - it: uses explicit sandboxNamespace for sandbox RoleBinding template: templates/rolebinding.yaml set: diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 4a602eae3..3d856ac4c 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -48,7 +48,7 @@ supervisor: # "combined" runs the current single supervisor container in the agent pod. # "sidecar" runs network enforcement in a dedicated sidecar and the process # supervisor as a low-capability wrapper in the agent container. - # "split-pod" runs network enforcement in a separate supervisor pod and + # "split-pod" runs network enforcement in a separate supervisor Deployment and # restricts the agent pod to that supervisor through NetworkPolicy. topology: "combined" # -- UID for the long-running network sidecar/split supervisor. In sidecar diff --git a/docs/kubernetes/setup.mdx b/docs/kubernetes/setup.mdx index bfbc0e0dc..359fae1a2 100644 --- a/docs/kubernetes/setup.mdx +++ b/docs/kubernetes/setup.mdx @@ -215,6 +215,10 @@ The namespaced Role covers sandbox lifecycle and identity: | `agents.x-k8s.io` | `sandboxes`, `sandboxes/status` | create, delete, get, list, patch, update, watch | | `""` | `events` | get, list, watch | | `""` | `pods` | get | +| `apps` | `deployments` | create, delete, get, list, watch | +| `apps` | `replicasets` | get | +| `""` | `services`, `secrets` | create, delete, get, list, watch | +| `networking.k8s.io` | `networkpolicies` | create, delete, get, list, watch | The ClusterRole grants node inspection and token validation: diff --git a/docs/kubernetes/topology.mdx b/docs/kubernetes/topology.mdx index ab9e3ce5a..1cc3b381c 100644 --- a/docs/kubernetes/topology.mdx +++ b/docs/kubernetes/topology.mdx @@ -99,20 +99,21 @@ dropping, or process/binary identity checks. ## Split-Pod Topology Split-pod topology moves network enforcement and gateway forwarding into a -separate supervisor pod. The agent pod runs the process supervisor and reaches -the supervisor through a per-sandbox headless Service. +separate supervisor Deployment with one pod. The agent pod runs the process +supervisor and reaches the supervisor through a per-sandbox headless Service. OpenShell creates these per-sandbox resources: - Agent pod labeled `openshell.ai/sandbox-role=agent`. -- Supervisor pod labeled `openshell.ai/sandbox-role=supervisor`. +- Supervisor Deployment with one pod labeled `openshell.ai/sandbox-role=supervisor`. - Headless Service for the supervisor pod. - Proxy CA Secret shared through mounts. - NetworkPolicy that limits agent egress to the supervisor pod and DNS. - NetworkPolicy that accepts supervisor ingress only from the paired agent pod. -The supervisor pod has a controlling `Sandbox` ownerReference so Kubernetes -garbage collection removes the companion resources when the sandbox is deleted. +The supervisor Deployment has a controlling `Sandbox` ownerReference so +Kubernetes garbage collection removes it when the sandbox is deleted. The +Deployment recreates the supervisor pod if the pod is deleted independently. Split-pod topology requires NetworkPolicy enforcement. In local k3s/k3d testing, @@ -175,7 +176,7 @@ supervisor: ``` The same `sidecarProxyUid` value is used as the non-root UID for the split -supervisor pod and must not match the sandbox UID. +supervisor pod created by the Deployment and must not match the sandbox UID. For direct gateway TOML configuration, set the equivalent Kubernetes driver fields: diff --git a/docs/reference/gateway-config.mdx b/docs/reference/gateway-config.mdx index 310d33107..6a5cc6a0a 100644 --- a/docs/reference/gateway-config.mdx +++ b/docs/reference/gateway-config.mdx @@ -181,7 +181,7 @@ supervisor_sideload_method = "image-volume" # filesystem, and network enforcement in the agent container. "sidecar" moves # pod-level network enforcement and gateway forwarding into a network sidecar. # "split-pod" moves network enforcement and gateway forwarding into a separate -# supervisor pod and uses NetworkPolicy to force agent egress through it. In +# supervisor Deployment and uses NetworkPolicy to force agent egress through it. In # sidecar and split-pod modes, the agent container runs non-root with no added # Linux capabilities and process/filesystem enforcement is network-only. supervisor_topology = "combined" diff --git a/docs/reference/sandbox-compute-drivers.mdx b/docs/reference/sandbox-compute-drivers.mdx index 8d8eab8aa..2517f65cf 100644 --- a/docs/reference/sandbox-compute-drivers.mdx +++ b/docs/reference/sandbox-compute-drivers.mdx @@ -304,7 +304,7 @@ For maintainer-level implementation details, refer to the [Kubernetes driver REA | `supervisor_image` | `supervisor.image.repository` / `supervisor.image.tag` | Set the supervisor image that provides the `openshell-sandbox` binary. | | `supervisor_image_pull_policy` | `supervisor.image.pullPolicy` | Set the Kubernetes image pull policy for the supervisor image. | | `supervisor_sideload_method` | `supervisor.sideloadMethod` | How the supervisor binary is delivered into sandbox pods. Leave empty to auto-detect from cluster version. Set to `image-volume` to mount the supervisor OCI image directly as a volume (requires Kubernetes 1.33+ with the ImageVolume feature gate; GA in 1.36), or `init-container` to copy it through an init container on older clusters. | -| `supervisor_topology` | `supervisor.topology` | Set `combined` for the default single supervisor path, `sidecar` to move pod-level network enforcement and gateway forwarding into a dedicated sidecar, or `split-pod` to run network enforcement in a separate supervisor pod with NetworkPolicy isolation. | +| `supervisor_topology` | `supervisor.topology` | Set `combined` for the default single supervisor path, `sidecar` to move pod-level network enforcement and gateway forwarding into a dedicated sidecar, or `split-pod` to run network enforcement in a separate supervisor Deployment with NetworkPolicy isolation. | | `sidecar_proxy_uid` | `supervisor.sidecarProxyUid` | UID used by the long-running network sidecar or split supervisor pod. In `sidecar` topology, the network init container exempts this UID from proxy redirection. | | `app_armor_profile` | `server.appArmorProfile` | Set the sandbox agent container's AppArmor profile. Helm defaults this to `Unconfined` so AppArmor-enabled nodes do not block supervisor network namespace setup. Set the Helm value to an empty string to omit the field, or use `RuntimeDefault` or `Localhost/` for operator-managed profiles. | | `workspace_default_storage_size` | `server.workspaceDefaultStorageSize` | Set the default workspace PVC size for new sandboxes. | @@ -316,10 +316,11 @@ policy, process privilege changes, and network policy enforcement. In `sidecar` and `split-pod` topology, the agent container runs as the resolved sandbox UID/GID with no added Linux capabilities. Sidecar mode uses a root init container for nftables setup and a long-running non-root sidecar. Split-pod mode -creates a separate non-root supervisor pod, a headless Service, a proxy CA -Secret, and per-sandbox NetworkPolicies. Both modes keep gateway session and SSH -behavior, but the process supervisor runs in network-only mode: filesystem -policy, process privilege dropping, and +creates a separate non-root supervisor Deployment with one pod, a headless +Service, a proxy CA Secret, and per-sandbox NetworkPolicies. The Deployment +recreates the supervisor pod if it is deleted. Both modes keep gateway session +and SSH behavior, but the process supervisor runs in network-only mode: +filesystem policy, process privilege dropping, and process/binary identity checks are not applied by the process container. Sidecar mode uses pod `fsGroup` so the non-root process supervisor can read the