diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index a2d5e21ee..71a31328e 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -107,8 +107,12 @@ log expressions can fail without rolling back the required table, chain, and reject rules. The agent container runs as the resolved sandbox UID/GID with no added Linux capabilities. Sidecar mode preserves gateway session and SSH behavior, but -treats the process leaf as network-only: Landlock filesystem policy, process -privilege dropping, and process/binary identity checks are not applied there. +treats the process leaf as network-only: Landlock filesystem policy and child +seccomp still apply where supported, while process privilege dropping and +supervisor identity mount isolation do not run because the agent container is +already unprivileged. Sidecar pods use a shared process namespace so the +network sidecar can resolve workload process and binary identity through +`/proc/`. ## Images diff --git a/crates/openshell-driver-kubernetes/README.md b/crates/openshell-driver-kubernetes/README.md index 8736e62a4..e9efc2867 100644 --- a/crates/openshell-driver-kubernetes/README.md +++ b/crates/openshell-driver-kubernetes/README.md @@ -62,15 +62,17 @@ container and runs the long-lived network sidecar as a non-root UID with no added Linux capabilities. The agent container also runs as the resolved sandbox UID/GID with `allowPrivilegeEscalation: false` and `capabilities.drop: ["ALL"]`. In this mode OpenShell preserves gateway session and SSH behavior, but the -process supervisor runs in network-only mode and does not apply Landlock -filesystem policy, process privilege dropping, or process/binary identity -checks. Network endpoint and L7 policy remain enforced by the network sidecar. +process supervisor does not perform root-to-sandbox privilege dropping or +supervisor identity mount isolation. It still applies Landlock filesystem policy +and child seccomp filters where the kernel/runtime supports them. Network +endpoint and L7 policy remain enforced by the network sidecar, and +sidecar pods use a shared process namespace so the network sidecar can resolve +process/binary identity through `/proc/`. -Sidecar mode uses the pod `fsGroup` to make the projected service-account token -and sandbox client TLS secret group-readable so the non-root process supervisor -can authenticate to the gateway. Treat the agent container as trusted with -respect to those in-pod gateway credentials until a narrower credential handoff -exists. +Sidecar mode keeps gateway credentials in the network sidecar. The agent +container does not mount the projected service-account token used for sandbox +token bootstrap, does not mount the sandbox client TLS secret, and does not get +gateway callback environment variables. The driver can request a Kubernetes AppArmor profile through `app_armor_profile`. diff --git a/crates/openshell-driver-kubernetes/src/config.rs b/crates/openshell-driver-kubernetes/src/config.rs index e33f15604..f07e4fb9a 100644 --- a/crates/openshell-driver-kubernetes/src/config.rs +++ b/crates/openshell-driver-kubernetes/src/config.rs @@ -88,6 +88,35 @@ impl FromStr for SupervisorTopology { } } +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(default, deny_unknown_fields)] +pub struct KubernetesSidecarConfig { + /// UID used by the long-running network sidecar in `sidecar` topology. + /// The network init container installs nftables rules that exempt this + /// UID, so it must not match the sandbox workload UID. + pub proxy_uid: u32, +} + +impl Default for KubernetesSidecarConfig { + fn default() -> Self { + Self { + proxy_uid: DEFAULT_PROXY_UID, + } + } +} + +impl KubernetesSidecarConfig { + pub fn validate_proxy_uid(&self) -> Result<(), String> { + if self.proxy_uid < openshell_policy::MIN_SANDBOX_UID { + return Err(format!( + "sidecar.proxy_uid must be at least {}", + openshell_policy::MIN_SANDBOX_UID + )); + } + Ok(()) + } +} + /// Kubernetes `AppArmor` profile requested for the sandbox agent container. #[derive(Debug, Clone, PartialEq, Eq)] pub enum AppArmorProfile { @@ -213,11 +242,9 @@ pub struct KubernetesComputeConfig { /// How the supervisor binary is delivered into sandbox pods. pub supervisor_sideload_method: SupervisorSideloadMethod, /// How the supervisor is arranged for Kubernetes sandbox pods. - pub supervisor_topology: SupervisorTopology, - /// UID used by the long-running network sidecar in `sidecar` topology. - /// The network init container installs nftables rules that exempt this - /// UID, so it must not match the sandbox workload UID. - pub proxy_uid: u32, + pub topology: SupervisorTopology, + /// Sidecar-only settings used when `topology = "sidecar"`. + pub sidecar: KubernetesSidecarConfig, pub grpc_endpoint: String, pub ssh_socket_path: String, pub client_tls_secret_name: String, @@ -303,8 +330,8 @@ impl Default for KubernetesComputeConfig { supervisor_image: DEFAULT_SUPERVISOR_IMAGE.to_string(), supervisor_image_pull_policy: String::new(), supervisor_sideload_method: SupervisorSideloadMethod::default(), - supervisor_topology: SupervisorTopology::default(), - proxy_uid: DEFAULT_PROXY_UID, + topology: SupervisorTopology::default(), + sidecar: KubernetesSidecarConfig::default(), grpc_endpoint: String::new(), ssh_socket_path: "/run/openshell/ssh.sock".to_string(), client_tls_secret_name: String::new(), @@ -350,13 +377,7 @@ impl KubernetesComputeConfig { } pub fn validate_proxy_uid(&self) -> Result<(), String> { - if self.proxy_uid < openshell_policy::MIN_SANDBOX_UID { - return Err(format!( - "proxy_uid must be at least {}", - openshell_policy::MIN_SANDBOX_UID - )); - } - Ok(()) + self.sidecar.validate_proxy_uid() } /// Resolve the sandbox UID/GID pair. @@ -486,50 +507,63 @@ mod tests { } #[test] - fn default_supervisor_topology_is_combined() { + fn default_topology_is_combined() { let cfg = KubernetesComputeConfig::default(); - assert_eq!(cfg.supervisor_topology, SupervisorTopology::Combined); - assert_eq!(cfg.supervisor_topology.to_string(), "combined"); + assert_eq!(cfg.topology, SupervisorTopology::Combined); + assert_eq!(cfg.topology.to_string(), "combined"); } #[test] fn default_proxy_uid_is_dedicated_non_root_uid() { let cfg = KubernetesComputeConfig::default(); - assert_eq!(cfg.proxy_uid, DEFAULT_PROXY_UID); + assert_eq!(cfg.sidecar.proxy_uid, DEFAULT_PROXY_UID); } #[test] - fn serde_override_supervisor_topology_sidecar() { + fn serde_override_topology_sidecar() { let json = serde_json::json!({ - "supervisor_topology": "sidecar" + "topology": "sidecar" }); let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap(); - assert_eq!(cfg.supervisor_topology, SupervisorTopology::Sidecar); + assert_eq!(cfg.topology, SupervisorTopology::Sidecar); } #[test] - fn serde_override_supervisor_topology_combined() { + fn serde_override_topology_combined() { let json = serde_json::json!({ - "supervisor_topology": "combined" + "topology": "combined" }); let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap(); - assert_eq!(cfg.supervisor_topology, SupervisorTopology::Combined); + assert_eq!(cfg.topology, SupervisorTopology::Combined); } #[test] - fn serde_override_proxy_uid() { + fn serde_rejects_sidecar_binary_identity_field() { let json = serde_json::json!({ - "proxy_uid": 2000 + "sidecar": { + "binary_identity": "shared-pid" + } + }); + let err = serde_json::from_value::(json).unwrap_err(); + assert!(err.to_string().contains("unknown field")); + } + + #[test] + fn serde_override_sidecar_proxy_uid_nested() { + let json = serde_json::json!({ + "sidecar": { + "proxy_uid": 2000 + } }); let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap(); - assert_eq!(cfg.proxy_uid, 2000); + assert_eq!(cfg.sidecar.proxy_uid, 2000); cfg.validate_proxy_uid().unwrap(); } #[test] fn validate_proxy_uid_rejects_privileged_uid() { let cfg = KubernetesComputeConfig { - proxy_uid: 999, + sidecar: KubernetesSidecarConfig { proxy_uid: 999 }, ..KubernetesComputeConfig::default() }; let err = cfg.validate_proxy_uid().unwrap_err(); @@ -537,14 +571,34 @@ mod tests { } #[test] - fn serde_rejects_invalid_supervisor_topology() { + fn serde_rejects_invalid_topology() { let json = serde_json::json!({ - "supervisor_topology": "unsupported" + "topology": "unsupported" }); let err = serde_json::from_value::(json).unwrap_err(); assert!(err.to_string().contains("unknown variant")); } + #[test] + fn serde_rejects_removed_supervisor_topology_field() { + let json = serde_json::json!({ + "supervisor_topology": "sidecar" + }); + let err = serde_json::from_value::(json).unwrap_err(); + assert!(err.to_string().contains("unknown field")); + } + + #[test] + fn serde_rejects_removed_flat_sidecar_fields() { + for json in [ + serde_json::json!({ "sidecar_binary_identity": "shared-pid" }), + serde_json::json!({ "proxy_uid": 2000 }), + ] { + let err = serde_json::from_value::(json).unwrap_err(); + assert!(err.to_string().contains("unknown field")); + } + } + #[test] fn serde_rejects_removed_process_enforcement_field() { let json = serde_json::json!({ diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index 8069927f2..568901255 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -563,8 +563,8 @@ impl KubernetesComputeDriver { supervisor_image: &self.config.supervisor_image, supervisor_image_pull_policy: &self.config.supervisor_image_pull_policy, supervisor_sideload_method: self.config.supervisor_sideload_method, - supervisor_topology: self.config.supervisor_topology, - proxy_uid: self.config.proxy_uid, + supervisor_topology: self.config.topology, + proxy_uid: self.config.sidecar.proxy_uid, service_account_name: &self.config.service_account_name, sandbox_id: &sandbox.id, sandbox_name: &sandbox.name, @@ -1334,11 +1334,6 @@ fn supervisor_sidecar_env( openshell_core::sandbox_env::NETWORK_ENFORCEMENT_MODE, "sidecar-nftables", ); - upsert_env( - &mut env, - openshell_core::sandbox_env::NETWORK_BINARY_IDENTITY, - "relaxed", - ); upsert_env( &mut env, openshell_core::sandbox_env::SUPERVISOR_READY_FILE, @@ -1488,6 +1483,8 @@ fn apply_supervisor_sidecar_topology( sc.insert("fsGroup".to_string(), serde_json::json!(params.sandbox_gid)); } + spec.insert("shareProcessNamespace".to_string(), serde_json::json!(true)); + apply_supervisor_binary_source( spec, params.supervisor_image, @@ -3124,10 +3121,7 @@ mod tests { ¶ms, ); - assert!( - pod_template["spec"]["shareProcessNamespace"].is_null(), - "sidecar mode no longer needs a shared process namespace when binary identity is relaxed" - ); + assert_eq!(pod_template["spec"]["shareProcessNamespace"], true); assert_eq!(pod_template["spec"]["securityContext"]["fsGroup"], 1500); let containers = pod_template["spec"]["containers"].as_array().unwrap(); assert_eq!(containers.len(), 2); @@ -3250,7 +3244,7 @@ mod tests { sidecar, openshell_core::sandbox_env::NETWORK_BINARY_IDENTITY ), - Some("relaxed") + None ); assert_eq!( rendered_env(sidecar, openshell_core::sandbox_env::ENTRYPOINT_PID_FILE), diff --git a/crates/openshell-driver-kubernetes/src/lib.rs b/crates/openshell-driver-kubernetes/src/lib.rs index 8c326f6af..7c56c8de5 100644 --- a/crates/openshell-driver-kubernetes/src/lib.rs +++ b/crates/openshell-driver-kubernetes/src/lib.rs @@ -7,8 +7,8 @@ pub mod grpc; pub use config::{ AppArmorProfile, DEFAULT_PROXY_UID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, - DEFAULT_WORKSPACE_STORAGE_SIZE, KubernetesComputeConfig, SupervisorSideloadMethod, - SupervisorTopology, + DEFAULT_WORKSPACE_STORAGE_SIZE, KubernetesComputeConfig, KubernetesSidecarConfig, + SupervisorSideloadMethod, SupervisorTopology, }; pub use driver::{KubernetesComputeDriver, KubernetesDriverError}; pub use grpc::ComputeDriverService; diff --git a/crates/openshell-driver-kubernetes/src/main.rs b/crates/openshell-driver-kubernetes/src/main.rs index 3271e56a4..8f6208766 100644 --- a/crates/openshell-driver-kubernetes/src/main.rs +++ b/crates/openshell-driver-kubernetes/src/main.rs @@ -11,7 +11,8 @@ use openshell_core::VERSION; use openshell_core::proto::compute::v1::compute_driver_server::ComputeDriverServer; use openshell_driver_kubernetes::{ AppArmorProfile, ComputeDriverService, DEFAULT_PROXY_UID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, - KubernetesComputeConfig, KubernetesComputeDriver, SupervisorSideloadMethod, SupervisorTopology, + KubernetesComputeConfig, KubernetesComputeDriver, KubernetesSidecarConfig, + SupervisorSideloadMethod, SupervisorTopology, }; #[derive(Parser, Debug)] @@ -82,13 +83,19 @@ struct Args { #[arg( long, - env = "OPENSHELL_SUPERVISOR_TOPOLOGY", + alias = "supervisor-topology", + env = "OPENSHELL_K8S_TOPOLOGY", default_value = "combined" )] - supervisor_topology: SupervisorTopology, + topology: SupervisorTopology, - #[arg(long, env = "OPENSHELL_PROXY_UID", default_value_t = DEFAULT_PROXY_UID)] - proxy_uid: u32, + #[arg( + long = "sidecar-proxy-uid", + alias = "proxy-uid", + env = "OPENSHELL_K8S_SIDECAR_PROXY_UID", + default_value_t = DEFAULT_PROXY_UID + )] + sidecar_proxy_uid: u32, #[arg(long, env = "OPENSHELL_ENABLE_USER_NAMESPACES")] enable_user_namespaces: bool, @@ -133,8 +140,10 @@ async fn main() -> Result<()> { .unwrap_or_else(|| openshell_core::config::DEFAULT_SUPERVISOR_IMAGE.to_string()), supervisor_image_pull_policy: args.supervisor_image_pull_policy.unwrap_or_default(), supervisor_sideload_method: args.supervisor_sideload_method, - supervisor_topology: args.supervisor_topology, - proxy_uid: args.proxy_uid, + topology: args.topology, + sidecar: KubernetesSidecarConfig { + proxy_uid: args.sidecar_proxy_uid, + }, grpc_endpoint: args.grpc_endpoint.unwrap_or_default(), ssh_socket_path: args.sandbox_ssh_socket_path, client_tls_secret_name: args.client_tls_secret_name.unwrap_or_default(), diff --git a/crates/openshell-supervisor-process/src/process.rs b/crates/openshell-supervisor-process/src/process.rs index bd8be04c8..72effa98f 100644 --- a/crates/openshell-supervisor-process/src/process.rs +++ b/crates/openshell-supervisor-process/src/process.rs @@ -42,9 +42,32 @@ pub enum ProcessEnforcementMode { impl ProcessEnforcementMode { #[must_use] - pub const fn enforces_process_controls(self) -> bool { + pub const fn uses_privileged_process_setup(self) -> bool { matches!(self, Self::Full) } + + #[must_use] + pub const fn enforces_child_sandbox(self) -> bool { + matches!(self, Self::Full | Self::NetworkOnly) + } +} + +#[cfg(target_os = "linux")] +pub(crate) fn prepare_child_sandbox( + policy: &SandboxPolicy, + workdir: Option<&str>, + enforcement_mode: ProcessEnforcementMode, +) -> Result> { + if !enforcement_mode.enforces_child_sandbox() { + return Ok(None); + } + + let prepared = if enforcement_mode.uses_privileged_process_setup() { + sandbox::linux::prepare(policy, workdir) + } else { + sandbox::linux::prepare_current_user(policy, workdir) + }?; + Ok(Some(prepared)) } const SUPERVISOR_ONLY_ENV_VARS: &[&str] = &[ @@ -580,24 +603,20 @@ impl ProcessHandle { // process where the tracing subscriber is functional. The child's // pre_exec context cannot reliably emit structured logs. #[cfg(target_os = "linux")] - if enforcement_mode.enforces_process_controls() { + if enforcement_mode.enforces_child_sandbox() { sandbox::linux::log_sandbox_readiness(policy, workdir); } - // Phase 1 (as root): Prepare Landlock ruleset by opening PathFds. - // This MUST happen before drop_privileges() so that root-only paths - // (e.g. mode 700 directories) can be opened. See issue #803. + // Phase 1: Prepare Landlock ruleset by opening PathFds. + // In full mode this runs before drop_privileges() so root-only paths + // can be opened. In sidecar network-only mode the container already + // runs as the sandbox UID, so inaccessible paths are unavailable to + // the workload and best-effort compatibility skips them. #[cfg(target_os = "linux")] - let prepared_sandbox = if enforcement_mode.enforces_process_controls() { - Some( - sandbox::linux::prepare(policy, workdir) - .map_err(|err| miette::miette!("Failed to prepare sandbox: {err}"))?, - ) - } else { - None - }; + let prepared_sandbox = prepare_child_sandbox(policy, workdir, enforcement_mode) + .map_err(|err| miette::miette!("Failed to prepare sandbox: {err}"))?; #[cfg(target_os = "linux")] - let supervisor_identity_mount = if enforcement_mode.enforces_process_controls() { + let supervisor_identity_mount = if enforcement_mode.uses_privileged_process_setup() { supervisor_identity_mount_from_env().map_err(|err| { miette::miette!("Failed to prepare supervisor identity isolation: {err}") })? @@ -640,7 +659,7 @@ impl ProcessHandle { // Drop privileges. initgroups/setgid/setuid need access to // /etc/group and /etc/passwd which would be blocked if // Landlock were already enforced. - if enforcement_mode.enforces_process_controls() { + if enforcement_mode.uses_privileged_process_setup() { drop_privileges(&policy) .map_err(|err| std::io::Error::other(err.to_string()))?; } @@ -741,14 +760,14 @@ impl ProcessHandle { // Drop privileges before applying sandbox restrictions. // initgroups/setgid/setuid need access to /etc/group and /etc/passwd // which may be blocked by Landlock. - if enforcement_mode.enforces_process_controls() { + if enforcement_mode.uses_privileged_process_setup() { drop_privileges(&policy) .map_err(|err| std::io::Error::other(err.to_string()))?; } harden_child_process().map_err(|err| std::io::Error::other(err.to_string()))?; - if enforcement_mode.enforces_process_controls() { + if enforcement_mode.enforces_child_sandbox() { sandbox::apply(&policy, workdir.as_deref()) .map_err(|err| std::io::Error::other(err.to_string()))?; } @@ -1504,6 +1523,18 @@ mod tests { ); } + #[test] + fn full_enforcement_uses_privileged_setup_and_child_sandbox() { + assert!(ProcessEnforcementMode::Full.uses_privileged_process_setup()); + assert!(ProcessEnforcementMode::Full.enforces_child_sandbox()); + } + + #[test] + fn network_only_enforcement_keeps_child_sandbox_without_privileged_setup() { + assert!(!ProcessEnforcementMode::NetworkOnly.uses_privileged_process_setup()); + assert!(ProcessEnforcementMode::NetworkOnly.enforces_child_sandbox()); + } + #[cfg(target_os = "linux")] fn capability_bounding_set_clear_available() -> bool { capctl::caps::CapState::get_current() diff --git a/crates/openshell-supervisor-process/src/run.rs b/crates/openshell-supervisor-process/src/run.rs index f7e25fd91..37f6ea77c 100644 --- a/crates/openshell-supervisor-process/src/run.rs +++ b/crates/openshell-supervisor-process/src/run.rs @@ -73,7 +73,7 @@ pub async fn run_process( // /etc/group so the "sandbox" entry matches. Must run before // validate_sandbox_user so passwd lookups see the correct identity. #[cfg(unix)] - if enforcement_mode.enforces_process_controls() { + if enforcement_mode.uses_privileged_process_setup() { crate::process::update_sandbox_passwd_entries()?; } @@ -81,7 +81,7 @@ pub async fn run_process( // must include a "sandbox" user for privilege dropping; failing fast here // beats silently running children as root. #[cfg(unix)] - if enforcement_mode.enforces_process_controls() { + if enforcement_mode.uses_privileged_process_setup() { crate::process::validate_sandbox_user(policy)?; crate::process::validate_sandbox_group(policy)?; } @@ -90,7 +90,7 @@ pub async fn run_process( // sandbox user/group. Runs as the supervisor (root) before the child // is forked so the workload sees writable paths it owns. #[cfg(unix)] - if enforcement_mode.enforces_process_controls() { + if enforcement_mode.uses_privileged_process_setup() { crate::process::prepare_filesystem(policy)?; } diff --git a/crates/openshell-supervisor-process/src/sandbox/linux/landlock.rs b/crates/openshell-supervisor-process/src/sandbox/linux/landlock.rs index 8808a1a87..222377b5d 100644 --- a/crates/openshell-supervisor-process/src/sandbox/linux/landlock.rs +++ b/crates/openshell-supervisor-process/src/sandbox/linux/landlock.rs @@ -5,7 +5,7 @@ use landlock::{ ABI, Access, AccessFs, CompatLevel, Compatible, PathBeneath, PathFd, PathFdError, Ruleset, - RulesetAttr, RulesetCreatedAttr, + RulesetAttr, }; use miette::{IntoDiagnostic, Result}; use openshell_core::policy::{LandlockCompatibility, SandboxPolicy}; @@ -95,6 +95,12 @@ pub struct PreparedRuleset { compatibility: LandlockCompatibility, } +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum PathOpenMode { + Privileged, + CurrentUser, +} + /// Phase 1: Open `PathFds` and build the Landlock ruleset **as root**. /// /// This must run before `drop_privileges()` so that `PathFd::new()` can open @@ -103,6 +109,27 @@ pub struct PreparedRuleset { /// Returns `None` if there are no filesystem paths to restrict (no-op). /// Returns `Some(PreparedRuleset)` on success, or an error. pub fn prepare(policy: &SandboxPolicy, workdir: Option<&str>) -> Result> { + prepare_with_path_open_mode(policy, workdir, PathOpenMode::Privileged) +} + +/// Phase 1 for already-unprivileged workloads. +/// +/// Kubernetes sidecar mode starts the process supervisor as the sandbox UID, so +/// Landlock path FDs are opened as the same UID that will run the workload. +/// Paths this UID cannot open are already unavailable to the workload; omit +/// them from the allowlist and let the resulting ruleset deny everything else. +pub fn prepare_current_user( + policy: &SandboxPolicy, + workdir: Option<&str>, +) -> Result> { + prepare_with_path_open_mode(policy, workdir, PathOpenMode::CurrentUser) +} + +fn prepare_with_path_open_mode( + policy: &SandboxPolicy, + workdir: Option<&str>, + path_open_mode: PathOpenMode, +) -> Result> { let read_only = policy.filesystem.read_only.clone(); let mut read_write = policy.filesystem.read_write.clone(); @@ -188,7 +215,7 @@ pub fn prepare(policy: &SandboxPolicy, workdir: Option<&str>) -> Result) -> Result) -> Result<()> { /// /// In `HardRequirement` mode, any failure is fatal — the caller propagates the /// error, which ultimately aborts sandbox startup. -fn try_open_path(path: &Path, compatibility: &LandlockCompatibility) -> Result> { +fn try_open_path( + path: &Path, + compatibility: &LandlockCompatibility, + path_open_mode: PathOpenMode, +) -> Result> { match PathFd::new(path) { Ok(fd) => Ok(Some(fd)), Err(err) => { @@ -335,6 +366,28 @@ fn try_open_path(path: &Path, compatibility: &LandlockCompatibility) -> Result { // NotFound is expected for stale baseline paths (e.g. @@ -421,6 +474,7 @@ mod tests { let result = try_open_path( &PathBuf::from("/nonexistent/openshell/test/path"), &LandlockCompatibility::BestEffort, + PathOpenMode::Privileged, ); assert!(result.is_ok()); assert!(result.unwrap().is_none()); @@ -431,6 +485,7 @@ mod tests { let result = try_open_path( &PathBuf::from("/nonexistent/openshell/test/path"), &LandlockCompatibility::HardRequirement, + PathOpenMode::Privileged, ); assert!(result.is_err()); let err_msg = result.unwrap_err().to_string(); @@ -447,11 +502,26 @@ mod tests { #[test] fn try_open_path_succeeds_for_existing_path() { let dir = tempfile::tempdir().unwrap(); - let result = try_open_path(dir.path(), &LandlockCompatibility::BestEffort); + let result = try_open_path( + dir.path(), + &LandlockCompatibility::BestEffort, + PathOpenMode::Privileged, + ); assert!(result.is_ok()); assert!(result.unwrap().is_some()); } + #[test] + fn try_open_path_current_user_skips_missing_path_in_hard_requirement() { + let result = try_open_path( + &PathBuf::from("/nonexistent/openshell/test/path"), + &LandlockCompatibility::HardRequirement, + PathOpenMode::CurrentUser, + ); + assert!(result.is_ok()); + assert!(result.unwrap().is_none()); + } + #[test] fn classify_not_found() { let err = std::io::Error::from_raw_os_error(libc::ENOENT); diff --git a/crates/openshell-supervisor-process/src/sandbox/linux/mod.rs b/crates/openshell-supervisor-process/src/sandbox/linux/mod.rs index b5397ef07..107a50e37 100644 --- a/crates/openshell-supervisor-process/src/sandbox/linux/mod.rs +++ b/crates/openshell-supervisor-process/src/sandbox/linux/mod.rs @@ -12,7 +12,7 @@ use std::path::PathBuf; use std::sync::Once; /// Opaque handle to a prepared-but-not-yet-enforced sandbox. -/// Holds the Landlock ruleset with `PathFds` opened as root. +/// Holds the Landlock ruleset with `PathFds` opened before child exec. pub struct PreparedSandbox { landlock: Option, policy: SandboxPolicy, @@ -30,6 +30,21 @@ pub fn prepare(policy: &SandboxPolicy, workdir: Option<&str>) -> Result, +) -> Result { + let landlock = landlock::prepare_current_user(policy, workdir)?; + Ok(PreparedSandbox { + landlock, + policy: policy.clone(), + }) +} + /// Phase 2: Enforce prepared sandbox restrictions (after `drop_privileges`). /// /// Calls `restrict_self()` for Landlock and applies seccomp filters. diff --git a/crates/openshell-supervisor-process/src/ssh.rs b/crates/openshell-supervisor-process/src/ssh.rs index 85466cb14..82118b841 100644 --- a/crates/openshell-supervisor-process/src/ssh.rs +++ b/crates/openshell-supervisor-process/src/ssh.rs @@ -65,7 +65,7 @@ fn ssh_server_init( if let Some(parent) = listen_path.parent() { std::fs::create_dir_all(parent).into_diagnostic()?; #[cfg(unix)] - if enforcement_mode.enforces_process_controls() && !shared_socket { + if enforcement_mode.uses_privileged_process_setup() && !shared_socket { use std::os::unix::fs::PermissionsExt; let perms = std::fs::Permissions::from_mode(0o700); std::fs::set_permissions(parent, perms).into_diagnostic()?; @@ -827,20 +827,15 @@ fn spawn_pty_shell( // Probe Landlock availability from the parent process where tracing works. #[cfg(target_os = "linux")] - if enforcement_mode.enforces_process_controls() { + if enforcement_mode.enforces_child_sandbox() { sandbox::linux::log_sandbox_readiness(policy, workdir.as_deref()); } - // Phase 1 (as root): Prepare Landlock ruleset before drop_privileges. + // Phase 1: Prepare Landlock ruleset before the child applies it. #[cfg(target_os = "linux")] - let prepared_sandbox = if enforcement_mode.enforces_process_controls() { - Some( - sandbox::linux::prepare(policy, workdir.as_deref()) - .map_err(|err| anyhow::anyhow!("Failed to prepare sandbox: {err}"))?, - ) - } else { - None - }; + let prepared_sandbox = + crate::process::prepare_child_sandbox(policy, workdir.as_deref(), enforcement_mode) + .map_err(|err| anyhow::anyhow!("Failed to prepare sandbox: {err}"))?; #[cfg(unix)] { @@ -986,20 +981,15 @@ fn spawn_pipe_exec( // Probe Landlock availability from the parent process where tracing works. #[cfg(target_os = "linux")] - if enforcement_mode.enforces_process_controls() { + if enforcement_mode.enforces_child_sandbox() { sandbox::linux::log_sandbox_readiness(policy, workdir.as_deref()); } - // Phase 1 (as root): Prepare Landlock ruleset before drop_privileges. + // Phase 1: Prepare Landlock ruleset before the child applies it. #[cfg(target_os = "linux")] - let prepared_sandbox = if enforcement_mode.enforces_process_controls() { - Some( - sandbox::linux::prepare(policy, workdir.as_deref()) - .map_err(|err| anyhow::anyhow!("Failed to prepare sandbox: {err}"))?, - ) - } else { - None - }; + let prepared_sandbox = + crate::process::prepare_child_sandbox(policy, workdir.as_deref(), enforcement_mode) + .map_err(|err| anyhow::anyhow!("Failed to prepare sandbox: {err}"))?; #[cfg(unix)] { @@ -1157,7 +1147,7 @@ mod unsafe_pty { #[cfg(target_os = "linux")] let mut prepared = prepared; #[cfg(target_os = "linux")] - let supervisor_identity_mount = if enforcement_mode.enforces_process_controls() { + let supervisor_identity_mount = if enforcement_mode.uses_privileged_process_setup() { crate::process::supervisor_identity_mount_from_env().map_err(|err| { anyhow::anyhow!("failed to prepare supervisor identity isolation: {err}") })? @@ -1205,7 +1195,7 @@ mod unsafe_pty { #[cfg(target_os = "linux")] let mut prepared = prepared; #[cfg(target_os = "linux")] - let supervisor_identity_mount = if enforcement_mode.enforces_process_controls() { + let supervisor_identity_mount = if enforcement_mode.uses_privileged_process_setup() { crate::process::supervisor_identity_mount_from_env().map_err(|err| { anyhow::anyhow!("failed to prepare supervisor identity isolation: {err}") })? @@ -1260,7 +1250,7 @@ mod unsafe_pty { // Drop privileges. initgroups/setgid/setuid need /etc/group and // /etc/passwd which would be blocked if Landlock were already enforced. - if enforcement_mode.enforces_process_controls() { + if enforcement_mode.uses_privileged_process_setup() { drop_privileges(policy).map_err(|err| std::io::Error::other(err.to_string()))?; } crate::process::harden_child_process() @@ -1275,7 +1265,7 @@ mod unsafe_pty { } #[cfg(not(target_os = "linux"))] - if enforcement_mode.enforces_process_controls() { + if enforcement_mode.enforces_child_sandbox() { sandbox::apply(policy, None).map_err(|err| std::io::Error::other(err.to_string()))?; } diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 4c7116539..6093edf63 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -237,7 +237,7 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | supervisor.image.pullPolicy | string | `""` | Supervisor image pull policy. Defaults to the gateway image pull policy when empty. | | supervisor.image.repository | string | `"ghcr.io/nvidia/openshell/supervisor"` | Supervisor image repository. | | supervisor.image.tag | string | `""` | Supervisor image tag. Defaults to the chart appVersion when empty. | -| supervisor.proxyUid | int | `1337` | UID for the long-running network sidecar in sidecar topology. The network init container installs nftables rules that exempt this UID. | +| supervisor.sidecar.proxyUid | int | `1337` | UID for the long-running network sidecar in sidecar topology. The network init container installs nftables rules that exempt this UID. | | supervisor.sideloadMethod | string | `""` | How the supervisor binary is delivered into sandbox pods. Empty (default) = auto-detect from cluster version: K8s >= v1.35 -> "image-volume" (ImageVolume enabled by default; GA in v1.36) K8s < v1.35 -> "init-container" (copies via init container + emptyDir) On K8s v1.33-v1.34 with the ImageVolume feature gate manually enabled, set this to "image-volume" explicitly. | | supervisor.topology | string | `"combined"` | Supervisor pod topology for Kubernetes sandboxes. "combined" runs the current single supervisor container in the agent pod. "sidecar" runs network enforcement in a dedicated sidecar and the process supervisor as a low-capability wrapper in the agent container. | | tolerations | list | `[]` | Tolerations for the gateway pod. | diff --git a/deploy/helm/openshell/templates/gateway-config.yaml b/deploy/helm/openshell/templates/gateway-config.yaml index 122ee3973..7ed16701d 100644 --- a/deploy/helm/openshell/templates/gateway-config.yaml +++ b/deploy/helm/openshell/templates/gateway-config.yaml @@ -113,8 +113,7 @@ data: grpc_endpoint = {{ include "openshell.grpcEndpoint" . | quote }} service_account_name = {{ include "openshell.sandboxServiceAccountName" . | quote }} supervisor_sideload_method = {{ include "openshell.supervisorSideloadMethod" . | quote }} - supervisor_topology = {{ .Values.supervisor.topology | default "combined" | quote }} - proxy_uid = {{ .Values.supervisor.proxyUid | default 1337 }} + topology = {{ .Values.supervisor.topology | default "combined" | quote }} sa_token_ttl_secs = {{ .Values.server.sandboxJwt.k8sSaTokenTtlSecs | default 3600 }} {{- if .Values.server.providerTokenGrants.spiffe.enabled }} provider_spiffe_workload_api_socket_path = {{ .Values.server.providerTokenGrants.spiffe.workloadApiSocketPath | quote }} @@ -143,3 +142,6 @@ data: {{- if .Values.supervisor.image.pullPolicy }} supervisor_image_pull_policy = {{ .Values.supervisor.image.pullPolicy | quote }} {{- end }} + + [openshell.drivers.kubernetes.sidecar] + proxy_uid = {{ .Values.supervisor.sidecar.proxyUid | default 1337 }} diff --git a/deploy/helm/openshell/tests/gateway_config_test.yaml b/deploy/helm/openshell/tests/gateway_config_test.yaml index 26453ed49..9e3dfd93f 100644 --- a/deploy/helm/openshell/tests/gateway_config_test.yaml +++ b/deploy/helm/openshell/tests/gateway_config_test.yaml @@ -83,23 +83,26 @@ tests: path: data["gateway.toml"] pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?service_account_name\s*=\s*"openshell-sandbox"' - - it: renders supervisor topology under [openshell.drivers.kubernetes] + - it: renders topology under [openshell.drivers.kubernetes] template: templates/gateway-config.yaml set: supervisor.topology: sidecar asserts: - matchRegex: path: data["gateway.toml"] - pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?supervisor_topology\s*=\s*"sidecar"' + pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?topology\s*=\s*"sidecar"' + - notMatchRegex: + path: data["gateway.toml"] + pattern: 'supervisor_topology\s*=' - - it: renders proxy uid under [openshell.drivers.kubernetes] + - it: renders proxy uid under [openshell.drivers.kubernetes.sidecar] template: templates/gateway-config.yaml set: - supervisor.proxyUid: 2200 + supervisor.sidecar.proxyUid: 2200 asserts: - matchRegex: path: data["gateway.toml"] - pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?proxy_uid\s*=\s*2200' + pattern: '(?ms)\[openshell\.drivers\.kubernetes\.sidecar\].*?proxy_uid\s*=\s*2200' - it: renders sandbox image pull secrets under [openshell.drivers.kubernetes] template: templates/gateway-config.yaml diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index d10204e08..b756142d8 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -49,9 +49,10 @@ supervisor: # "sidecar" runs network enforcement in a dedicated sidecar and the process # supervisor as a low-capability wrapper in the agent container. topology: "combined" - # -- UID for the long-running network sidecar in sidecar topology. The - # network init container installs nftables rules that exempt this UID. - proxyUid: 1337 + sidecar: + # -- UID for the long-running network sidecar in sidecar topology. The + # network init container installs nftables rules that exempt this UID. + proxyUid: 1337 # -- Image pull secrets attached to gateway and helper pods. imagePullSecrets: [] diff --git a/docs/kubernetes/setup.mdx b/docs/kubernetes/setup.mdx index cc886c168..d97200b19 100644 --- a/docs/kubernetes/setup.mdx +++ b/docs/kubernetes/setup.mdx @@ -161,7 +161,7 @@ The most commonly changed values are: | `pkiInitJob.serverDnsNames` / `certManager.serverDnsNames` | Additional gateway server DNS SANs. Wildcard SANs also enable sandbox service URLs under that domain. | | `supervisor.sideloadMethod` | How the supervisor binary is delivered into sandbox pods. Leave empty to auto-detect based on cluster version: clusters running Kubernetes 1.35 or later use `image-volume` (ImageVolume GA in 1.36); older clusters use `init-container`. Set explicitly to `image-volume` on Kubernetes 1.33 or 1.34 with the ImageVolume feature gate enabled, or to `init-container` to force the legacy path on any version. | | `supervisor.topology` | Sandbox pod topology. Refer to [Topology](/kubernetes/topology). | -| `supervisor.proxyUid` | Non-root UID for the long-running network sidecar when `supervisor.topology=sidecar`. The UID must not match the sandbox UID. | +| `supervisor.sidecar.proxyUid` | Non-root UID for the long-running network sidecar when `supervisor.topology=sidecar`. The UID must not match the sandbox UID. | Use a values file for repeatable deployments: diff --git a/docs/kubernetes/topology.mdx b/docs/kubernetes/topology.mdx index c2e13c5c6..716141596 100644 --- a/docs/kubernetes/topology.mdx +++ b/docs/kubernetes/topology.mdx @@ -21,7 +21,7 @@ lower-privilege agent container. | Topology | Use when | Main tradeoff | |---|---|---| | `combined` | You need OpenShell network, filesystem, and process controls in the sandbox workload. | The agent container carries the Linux capabilities the supervisor needs. | -| `sidecar` | You need the agent container to run as non-root without added Linux capabilities, and network policy is the primary control. | The process supervisor is network-only, so filesystem/process controls do not run in the agent container. | +| `sidecar` | You need the agent container to run as non-root without added Linux capabilities, and network policy is the primary control. | Privilege-dropping and supervisor mount isolation do not run in the agent container. | ## Privilege Model @@ -121,7 +121,7 @@ The pod contains these OpenShell-managed pieces: | Component | Runs as | Purpose | |---|---|---| | Network init container | Root with setup capabilities | Installs pod-level nftables rules and prepares shared sidecar state. | -| Network sidecar | `supervisor.proxyUid` | Runs the proxy, enforces network policy, owns gateway authentication and the gateway session, and writes proxy TLS plus local policy/provider snapshots. | +| Network sidecar | `supervisor.sidecar.proxyUid` | Runs the proxy, enforces network policy, owns gateway authentication and the gateway session, and writes proxy TLS plus local policy/provider snapshots. | | Agent container | Resolved sandbox UID/GID | Runs the process supervisor and launches the user workload. | In this topology, the agent container defaults to `runAsNonRoot: true`, @@ -137,10 +137,14 @@ default sidecar path. Sidecar mode runs the process supervisor in `network-only` mode. OpenShell still -enforces network endpoint and L7 policy through the sidecar, but the process -supervisor does not apply Landlock filesystem policy, process privilege -dropping, or process/binary identity checks. Use `combined` topology when you -need those controls in the same supervisor path. +enforces network endpoint and L7 policy through the sidecar, and the process +supervisor applies Landlock filesystem policy and child seccomp filters where +the kernel/runtime supports them. The process supervisor does not perform +root-to-sandbox privilege dropping because Kubernetes starts the container as +the sandbox UID/GID, and it does not perform supervisor identity mount +isolation because gateway credentials are not mounted into the agent container. +Sidecar pods use `shareProcessNamespace: true` so the network sidecar can +resolve workload process and binary identity through `/proc/`. ## Credential Exposure @@ -161,9 +165,9 @@ watches for newer revisions that the network sidecar writes after settings polls, so future child processes can see refreshed provider env without giving the agent container gateway authentication material. This does not mutate the environment of the already-running workload entrypoint. Use `combined` topology -when you need full process/filesystem enforcement in the same supervisor path; -use additional runtime isolation when you need a stronger container boundary -around network-only sidecar workloads. +when you need the full single-supervisor enforcement path; use additional +runtime isolation when you need a stronger container boundary around sidecar +workloads. ## RuntimeClass Isolation @@ -172,9 +176,10 @@ Containers when the cluster supports them. A sandboxed runtime strengthens the container boundary while OpenShell focuses on network policy enforcement from the sidecar. -Runtime classes do not re-enable the OpenShell filesystem and process controls -that sidecar mode relaxes. Use them as an additional workload boundary, not as a -replacement for the combined topology's full supervisor controls. +Runtime classes do not re-enable the OpenShell privilege-drop or supervisor +mount-isolation controls that sidecar mode relaxes. Use them as an additional +workload boundary, not as a replacement for the combined topology's full +supervisor controls. You can set a default runtime class in the Kubernetes driver configuration or override it per sandbox with driver config: @@ -191,7 +196,9 @@ For direct gateway TOML configuration, set the Kubernetes driver fields: ```toml [openshell.drivers.kubernetes] -supervisor_topology = "sidecar" +topology = "sidecar" + +[openshell.drivers.kubernetes.sidecar] proxy_uid = 1337 ``` @@ -204,11 +211,12 @@ When the Helm chart renders `gateway.toml`, set the equivalent chart values: ```yaml supervisor: topology: sidecar - proxyUid: 1337 + sidecar: + proxyUid: 1337 ``` -Leave `supervisor_topology` unset, or set it to `combined`, to keep the -original single-container supervisor path. For Helm installs, leave +Leave `topology` unset, or set it to `combined`, to keep the original +single-container supervisor path. For Helm installs, leave `supervisor.topology` unset or set it to `combined`. ## Next Steps diff --git a/docs/reference/gateway-config.mdx b/docs/reference/gateway-config.mdx index eb3a23566..0e190b342 100644 --- a/docs/reference/gateway-config.mdx +++ b/docs/reference/gateway-config.mdx @@ -180,10 +180,7 @@ supervisor_sideload_method = "image-volume" # "combined" runs the existing single supervisor container with full process, # filesystem, and network enforcement in the agent container. "sidecar" moves # pod-level network enforcement and gateway session handling into a network sidecar. -supervisor_topology = "combined" -# UID used by the long-running network sidecar. In sidecar topology the -# network init container installs nftables rules that exempt this UID. -proxy_uid = 1337 +topology = "combined" grpc_endpoint = "https://openshell-gateway.agents.svc:8080" ssh_socket_path = "/run/openshell/ssh.sock" client_tls_secret_name = "openshell-client-tls" @@ -207,6 +204,11 @@ provider_spiffe_workload_api_socket_path = "/spiffe-workload-api/spire-agent.soc # back to 1000 on non-OpenShift clusters. # sandbox_uid = 1500 # sandbox_gid = 1500 + +[openshell.drivers.kubernetes.sidecar] +# UID used by the long-running network sidecar. In sidecar topology the +# network init container installs nftables rules that exempt this UID. +proxy_uid = 1337 ``` ### Docker diff --git a/docs/reference/sandbox-compute-drivers.mdx b/docs/reference/sandbox-compute-drivers.mdx index b2cee8154..71e76900d 100644 --- a/docs/reference/sandbox-compute-drivers.mdx +++ b/docs/reference/sandbox-compute-drivers.mdx @@ -306,8 +306,8 @@ For maintainer-level implementation details, refer to the [Kubernetes driver REA | `supervisor_image` | `supervisor.image.repository` / `supervisor.image.tag` | Set the supervisor image that provides the `openshell-sandbox` binary. | | `supervisor_image_pull_policy` | `supervisor.image.pullPolicy` | Set the Kubernetes image pull policy for the supervisor image. | | `supervisor_sideload_method` | `supervisor.sideloadMethod` | How the supervisor binary is delivered into sandbox pods. Leave empty to auto-detect from cluster version. Set to `image-volume` to mount the supervisor OCI image directly as a volume (requires Kubernetes 1.33+ with the ImageVolume feature gate; GA in 1.36), or `init-container` to copy it through an init container on older clusters. | -| `supervisor_topology` | `supervisor.topology` | Set `combined` for the default single supervisor path, or `sidecar` to move pod-level network enforcement and the gateway session into a dedicated sidecar. | -| `proxy_uid` | `supervisor.proxyUid` | UID used by the long-running network sidecar in `sidecar` topology. The network init container exempts this UID from proxy redirection. | +| `topology` | `supervisor.topology` | Set `combined` for the default single supervisor path, or `sidecar` to move pod-level network enforcement and the gateway session into a dedicated sidecar. | +| `sidecar.proxy_uid` | `supervisor.sidecar.proxyUid` | UID used by the long-running network sidecar in `sidecar` topology. The network init container exempts this UID from proxy redirection. | | `app_armor_profile` | `server.appArmorProfile` | Set the sandbox agent container's AppArmor profile. Helm defaults this to `Unconfined` so AppArmor-enabled nodes do not block supervisor network namespace setup. Set the Helm value to an empty string to omit the field, or use `RuntimeDefault` or `Localhost/` for operator-managed profiles. | | `workspace_default_storage_size` | `server.workspaceDefaultStorageSize` | Set the default workspace PVC size for new sandboxes. | | `sa_token_ttl_secs` | `server.sandboxJwt.k8sSaTokenTtlSecs` | Set the projected ServiceAccount token TTL used for the bootstrap token exchange. | @@ -324,10 +324,12 @@ sandbox bootstrap token or client TLS secret in the default sidecar path. The provider environment snapshot is refreshed by the network sidecar after settings polls so future child processes can see updated provider env without gateway access in the agent container. -Sidecar mode keeps gateway session and SSH behavior, but the process supervisor -runs in `network-only` mode: filesystem policy, process privilege dropping, and -process/binary identity checks are not applied by the process container. Network -policy still runs in the sidecar and remains endpoint/L7 scoped. +Sidecar mode keeps gateway session and SSH behavior. The process supervisor +applies Landlock filesystem policy and child seccomp filters where supported, +but it does not perform root-to-sandbox privilege dropping or supervisor +identity mount isolation. Network policy still runs in the sidecar, and sidecar +pods set `shareProcessNamespace: true` so the network sidecar can resolve +process/binary identity through `/proc/`. The Kubernetes driver creates namespaced `agents.x-k8s.io` `Sandbox` resources from the Kubernetes SIG Apps [agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) project. It detects the served Sandbox API at runtime, caches the selected API version for the gateway process, and uses `v1beta1` when available before falling back to `v1alpha1`, so supported Agent Sandbox installations work without version-specific operator configuration. The Agent Sandbox controller turns those resources into sandbox pods and related storage.