fix(kubernetes): preserve process identity in sidecar topology

Render sidecar pods with a shared process namespace, keep binary-aware network policy enabled, and move Kubernetes sidecar settings under the nested sidecar config table.

Also apply unprivileged Landlock/seccomp setup in NetworkOnly supervisor mode so sidecar topology keeps sandbox child hardening without privileged process setup.

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
This commit is contained in:
Taylor Mutch
2026-07-08 12:12:35 -07:00
parent 5681187871
commit cd0477fb23
19 changed files with 337 additions and 150 deletions
+6 -2
View File
@@ -107,8 +107,12 @@ log expressions can fail without rolling back the required table, chain, and
reject rules.
The agent container runs as the resolved sandbox UID/GID with no added Linux
capabilities. Sidecar mode preserves gateway session and SSH behavior, but
treats the process leaf as network-only: Landlock filesystem policy, process
privilege dropping, and process/binary identity checks are not applied there.
treats the process leaf as network-only: Landlock filesystem policy and child
seccomp still apply where supported, while process privilege dropping and
supervisor identity mount isolation do not run because the agent container is
already unprivileged. Sidecar pods use a shared process namespace so the
network sidecar can resolve workload process and binary identity through
`/proc/<entrypoint-pid>`.
## Images
+10 -8
View File
@@ -62,15 +62,17 @@ container and runs the long-lived network sidecar as a non-root UID with no
added Linux capabilities. The agent container also runs as the resolved sandbox
UID/GID with `allowPrivilegeEscalation: false` and `capabilities.drop: ["ALL"]`.
In this mode OpenShell preserves gateway session and SSH behavior, but the
process supervisor runs in network-only mode and does not apply Landlock
filesystem policy, process privilege dropping, or process/binary identity
checks. Network endpoint and L7 policy remain enforced by the network sidecar.
process supervisor does not perform root-to-sandbox privilege dropping or
supervisor identity mount isolation. It still applies Landlock filesystem policy
and child seccomp filters where the kernel/runtime supports them. Network
endpoint and L7 policy remain enforced by the network sidecar, and
sidecar pods use a shared process namespace so the network sidecar can resolve
process/binary identity through `/proc/<entrypoint-pid>`.
Sidecar mode uses the pod `fsGroup` to make the projected service-account token
and sandbox client TLS secret group-readable so the non-root process supervisor
can authenticate to the gateway. Treat the agent container as trusted with
respect to those in-pod gateway credentials until a narrower credential handoff
exists.
Sidecar mode keeps gateway credentials in the network sidecar. The agent
container does not mount the projected service-account token used for sandbox
token bootstrap, does not mount the sandbox client TLS secret, and does not get
gateway callback environment variables.
The driver can request a Kubernetes AppArmor profile through
`app_armor_profile`.
@@ -88,6 +88,35 @@ impl FromStr for SupervisorTopology {
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct KubernetesSidecarConfig {
/// UID used by the long-running network sidecar in `sidecar` topology.
/// The network init container installs nftables rules that exempt this
/// UID, so it must not match the sandbox workload UID.
pub proxy_uid: u32,
}
impl Default for KubernetesSidecarConfig {
fn default() -> Self {
Self {
proxy_uid: DEFAULT_PROXY_UID,
}
}
}
impl KubernetesSidecarConfig {
pub fn validate_proxy_uid(&self) -> Result<(), String> {
if self.proxy_uid < openshell_policy::MIN_SANDBOX_UID {
return Err(format!(
"sidecar.proxy_uid must be at least {}",
openshell_policy::MIN_SANDBOX_UID
));
}
Ok(())
}
}
/// Kubernetes `AppArmor` profile requested for the sandbox agent container.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum AppArmorProfile {
@@ -213,11 +242,9 @@ pub struct KubernetesComputeConfig {
/// How the supervisor binary is delivered into sandbox pods.
pub supervisor_sideload_method: SupervisorSideloadMethod,
/// How the supervisor is arranged for Kubernetes sandbox pods.
pub supervisor_topology: SupervisorTopology,
/// UID used by the long-running network sidecar in `sidecar` topology.
/// The network init container installs nftables rules that exempt this
/// UID, so it must not match the sandbox workload UID.
pub proxy_uid: u32,
pub topology: SupervisorTopology,
/// Sidecar-only settings used when `topology = "sidecar"`.
pub sidecar: KubernetesSidecarConfig,
pub grpc_endpoint: String,
pub ssh_socket_path: String,
pub client_tls_secret_name: String,
@@ -303,8 +330,8 @@ impl Default for KubernetesComputeConfig {
supervisor_image: DEFAULT_SUPERVISOR_IMAGE.to_string(),
supervisor_image_pull_policy: String::new(),
supervisor_sideload_method: SupervisorSideloadMethod::default(),
supervisor_topology: SupervisorTopology::default(),
proxy_uid: DEFAULT_PROXY_UID,
topology: SupervisorTopology::default(),
sidecar: KubernetesSidecarConfig::default(),
grpc_endpoint: String::new(),
ssh_socket_path: "/run/openshell/ssh.sock".to_string(),
client_tls_secret_name: String::new(),
@@ -350,13 +377,7 @@ impl KubernetesComputeConfig {
}
pub fn validate_proxy_uid(&self) -> Result<(), String> {
if self.proxy_uid < openshell_policy::MIN_SANDBOX_UID {
return Err(format!(
"proxy_uid must be at least {}",
openshell_policy::MIN_SANDBOX_UID
));
}
Ok(())
self.sidecar.validate_proxy_uid()
}
/// Resolve the sandbox UID/GID pair.
@@ -486,50 +507,63 @@ mod tests {
}
#[test]
fn default_supervisor_topology_is_combined() {
fn default_topology_is_combined() {
let cfg = KubernetesComputeConfig::default();
assert_eq!(cfg.supervisor_topology, SupervisorTopology::Combined);
assert_eq!(cfg.supervisor_topology.to_string(), "combined");
assert_eq!(cfg.topology, SupervisorTopology::Combined);
assert_eq!(cfg.topology.to_string(), "combined");
}
#[test]
fn default_proxy_uid_is_dedicated_non_root_uid() {
let cfg = KubernetesComputeConfig::default();
assert_eq!(cfg.proxy_uid, DEFAULT_PROXY_UID);
assert_eq!(cfg.sidecar.proxy_uid, DEFAULT_PROXY_UID);
}
#[test]
fn serde_override_supervisor_topology_sidecar() {
fn serde_override_topology_sidecar() {
let json = serde_json::json!({
"supervisor_topology": "sidecar"
"topology": "sidecar"
});
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
assert_eq!(cfg.supervisor_topology, SupervisorTopology::Sidecar);
assert_eq!(cfg.topology, SupervisorTopology::Sidecar);
}
#[test]
fn serde_override_supervisor_topology_combined() {
fn serde_override_topology_combined() {
let json = serde_json::json!({
"supervisor_topology": "combined"
"topology": "combined"
});
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
assert_eq!(cfg.supervisor_topology, SupervisorTopology::Combined);
assert_eq!(cfg.topology, SupervisorTopology::Combined);
}
#[test]
fn serde_override_proxy_uid() {
fn serde_rejects_sidecar_binary_identity_field() {
let json = serde_json::json!({
"proxy_uid": 2000
"sidecar": {
"binary_identity": "shared-pid"
}
});
let err = serde_json::from_value::<KubernetesComputeConfig>(json).unwrap_err();
assert!(err.to_string().contains("unknown field"));
}
#[test]
fn serde_override_sidecar_proxy_uid_nested() {
let json = serde_json::json!({
"sidecar": {
"proxy_uid": 2000
}
});
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
assert_eq!(cfg.proxy_uid, 2000);
assert_eq!(cfg.sidecar.proxy_uid, 2000);
cfg.validate_proxy_uid().unwrap();
}
#[test]
fn validate_proxy_uid_rejects_privileged_uid() {
let cfg = KubernetesComputeConfig {
proxy_uid: 999,
sidecar: KubernetesSidecarConfig { proxy_uid: 999 },
..KubernetesComputeConfig::default()
};
let err = cfg.validate_proxy_uid().unwrap_err();
@@ -537,14 +571,34 @@ mod tests {
}
#[test]
fn serde_rejects_invalid_supervisor_topology() {
fn serde_rejects_invalid_topology() {
let json = serde_json::json!({
"supervisor_topology": "unsupported"
"topology": "unsupported"
});
let err = serde_json::from_value::<KubernetesComputeConfig>(json).unwrap_err();
assert!(err.to_string().contains("unknown variant"));
}
#[test]
fn serde_rejects_removed_supervisor_topology_field() {
let json = serde_json::json!({
"supervisor_topology": "sidecar"
});
let err = serde_json::from_value::<KubernetesComputeConfig>(json).unwrap_err();
assert!(err.to_string().contains("unknown field"));
}
#[test]
fn serde_rejects_removed_flat_sidecar_fields() {
for json in [
serde_json::json!({ "sidecar_binary_identity": "shared-pid" }),
serde_json::json!({ "proxy_uid": 2000 }),
] {
let err = serde_json::from_value::<KubernetesComputeConfig>(json).unwrap_err();
assert!(err.to_string().contains("unknown field"));
}
}
#[test]
fn serde_rejects_removed_process_enforcement_field() {
let json = serde_json::json!({
@@ -563,8 +563,8 @@ impl KubernetesComputeDriver {
supervisor_image: &self.config.supervisor_image,
supervisor_image_pull_policy: &self.config.supervisor_image_pull_policy,
supervisor_sideload_method: self.config.supervisor_sideload_method,
supervisor_topology: self.config.supervisor_topology,
proxy_uid: self.config.proxy_uid,
supervisor_topology: self.config.topology,
proxy_uid: self.config.sidecar.proxy_uid,
service_account_name: &self.config.service_account_name,
sandbox_id: &sandbox.id,
sandbox_name: &sandbox.name,
@@ -1334,11 +1334,6 @@ fn supervisor_sidecar_env(
openshell_core::sandbox_env::NETWORK_ENFORCEMENT_MODE,
"sidecar-nftables",
);
upsert_env(
&mut env,
openshell_core::sandbox_env::NETWORK_BINARY_IDENTITY,
"relaxed",
);
upsert_env(
&mut env,
openshell_core::sandbox_env::SUPERVISOR_READY_FILE,
@@ -1488,6 +1483,8 @@ fn apply_supervisor_sidecar_topology(
sc.insert("fsGroup".to_string(), serde_json::json!(params.sandbox_gid));
}
spec.insert("shareProcessNamespace".to_string(), serde_json::json!(true));
apply_supervisor_binary_source(
spec,
params.supervisor_image,
@@ -3124,10 +3121,7 @@ mod tests {
&params,
);
assert!(
pod_template["spec"]["shareProcessNamespace"].is_null(),
"sidecar mode no longer needs a shared process namespace when binary identity is relaxed"
);
assert_eq!(pod_template["spec"]["shareProcessNamespace"], true);
assert_eq!(pod_template["spec"]["securityContext"]["fsGroup"], 1500);
let containers = pod_template["spec"]["containers"].as_array().unwrap();
assert_eq!(containers.len(), 2);
@@ -3250,7 +3244,7 @@ mod tests {
sidecar,
openshell_core::sandbox_env::NETWORK_BINARY_IDENTITY
),
Some("relaxed")
None
);
assert_eq!(
rendered_env(sidecar, openshell_core::sandbox_env::ENTRYPOINT_PID_FILE),
@@ -7,8 +7,8 @@ pub mod grpc;
pub use config::{
AppArmorProfile, DEFAULT_PROXY_UID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME,
DEFAULT_WORKSPACE_STORAGE_SIZE, KubernetesComputeConfig, SupervisorSideloadMethod,
SupervisorTopology,
DEFAULT_WORKSPACE_STORAGE_SIZE, KubernetesComputeConfig, KubernetesSidecarConfig,
SupervisorSideloadMethod, SupervisorTopology,
};
pub use driver::{KubernetesComputeDriver, KubernetesDriverError};
pub use grpc::ComputeDriverService;
+16 -7
View File
@@ -11,7 +11,8 @@ use openshell_core::VERSION;
use openshell_core::proto::compute::v1::compute_driver_server::ComputeDriverServer;
use openshell_driver_kubernetes::{
AppArmorProfile, ComputeDriverService, DEFAULT_PROXY_UID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME,
KubernetesComputeConfig, KubernetesComputeDriver, SupervisorSideloadMethod, SupervisorTopology,
KubernetesComputeConfig, KubernetesComputeDriver, KubernetesSidecarConfig,
SupervisorSideloadMethod, SupervisorTopology,
};
#[derive(Parser, Debug)]
@@ -82,13 +83,19 @@ struct Args {
#[arg(
long,
env = "OPENSHELL_SUPERVISOR_TOPOLOGY",
alias = "supervisor-topology",
env = "OPENSHELL_K8S_TOPOLOGY",
default_value = "combined"
)]
supervisor_topology: SupervisorTopology,
topology: SupervisorTopology,
#[arg(long, env = "OPENSHELL_PROXY_UID", default_value_t = DEFAULT_PROXY_UID)]
proxy_uid: u32,
#[arg(
long = "sidecar-proxy-uid",
alias = "proxy-uid",
env = "OPENSHELL_K8S_SIDECAR_PROXY_UID",
default_value_t = DEFAULT_PROXY_UID
)]
sidecar_proxy_uid: u32,
#[arg(long, env = "OPENSHELL_ENABLE_USER_NAMESPACES")]
enable_user_namespaces: bool,
@@ -133,8 +140,10 @@ async fn main() -> Result<()> {
.unwrap_or_else(|| openshell_core::config::DEFAULT_SUPERVISOR_IMAGE.to_string()),
supervisor_image_pull_policy: args.supervisor_image_pull_policy.unwrap_or_default(),
supervisor_sideload_method: args.supervisor_sideload_method,
supervisor_topology: args.supervisor_topology,
proxy_uid: args.proxy_uid,
topology: args.topology,
sidecar: KubernetesSidecarConfig {
proxy_uid: args.sidecar_proxy_uid,
},
grpc_endpoint: args.grpc_endpoint.unwrap_or_default(),
ssh_socket_path: args.sandbox_ssh_socket_path,
client_tls_secret_name: args.client_tls_secret_name.unwrap_or_default(),
@@ -42,9 +42,32 @@ pub enum ProcessEnforcementMode {
impl ProcessEnforcementMode {
#[must_use]
pub const fn enforces_process_controls(self) -> bool {
pub const fn uses_privileged_process_setup(self) -> bool {
matches!(self, Self::Full)
}
#[must_use]
pub const fn enforces_child_sandbox(self) -> bool {
matches!(self, Self::Full | Self::NetworkOnly)
}
}
#[cfg(target_os = "linux")]
pub(crate) fn prepare_child_sandbox(
policy: &SandboxPolicy,
workdir: Option<&str>,
enforcement_mode: ProcessEnforcementMode,
) -> Result<Option<sandbox::linux::PreparedSandbox>> {
if !enforcement_mode.enforces_child_sandbox() {
return Ok(None);
}
let prepared = if enforcement_mode.uses_privileged_process_setup() {
sandbox::linux::prepare(policy, workdir)
} else {
sandbox::linux::prepare_current_user(policy, workdir)
}?;
Ok(Some(prepared))
}
const SUPERVISOR_ONLY_ENV_VARS: &[&str] = &[
@@ -580,24 +603,20 @@ impl ProcessHandle {
// process where the tracing subscriber is functional. The child's
// pre_exec context cannot reliably emit structured logs.
#[cfg(target_os = "linux")]
if enforcement_mode.enforces_process_controls() {
if enforcement_mode.enforces_child_sandbox() {
sandbox::linux::log_sandbox_readiness(policy, workdir);
}
// Phase 1 (as root): Prepare Landlock ruleset by opening PathFds.
// This MUST happen before drop_privileges() so that root-only paths
// (e.g. mode 700 directories) can be opened. See issue #803.
// Phase 1: Prepare Landlock ruleset by opening PathFds.
// In full mode this runs before drop_privileges() so root-only paths
// can be opened. In sidecar network-only mode the container already
// runs as the sandbox UID, so inaccessible paths are unavailable to
// the workload and best-effort compatibility skips them.
#[cfg(target_os = "linux")]
let prepared_sandbox = if enforcement_mode.enforces_process_controls() {
Some(
sandbox::linux::prepare(policy, workdir)
.map_err(|err| miette::miette!("Failed to prepare sandbox: {err}"))?,
)
} else {
None
};
let prepared_sandbox = prepare_child_sandbox(policy, workdir, enforcement_mode)
.map_err(|err| miette::miette!("Failed to prepare sandbox: {err}"))?;
#[cfg(target_os = "linux")]
let supervisor_identity_mount = if enforcement_mode.enforces_process_controls() {
let supervisor_identity_mount = if enforcement_mode.uses_privileged_process_setup() {
supervisor_identity_mount_from_env().map_err(|err| {
miette::miette!("Failed to prepare supervisor identity isolation: {err}")
})?
@@ -640,7 +659,7 @@ impl ProcessHandle {
// Drop privileges. initgroups/setgid/setuid need access to
// /etc/group and /etc/passwd which would be blocked if
// Landlock were already enforced.
if enforcement_mode.enforces_process_controls() {
if enforcement_mode.uses_privileged_process_setup() {
drop_privileges(&policy)
.map_err(|err| std::io::Error::other(err.to_string()))?;
}
@@ -741,14 +760,14 @@ impl ProcessHandle {
// Drop privileges before applying sandbox restrictions.
// initgroups/setgid/setuid need access to /etc/group and /etc/passwd
// which may be blocked by Landlock.
if enforcement_mode.enforces_process_controls() {
if enforcement_mode.uses_privileged_process_setup() {
drop_privileges(&policy)
.map_err(|err| std::io::Error::other(err.to_string()))?;
}
harden_child_process().map_err(|err| std::io::Error::other(err.to_string()))?;
if enforcement_mode.enforces_process_controls() {
if enforcement_mode.enforces_child_sandbox() {
sandbox::apply(&policy, workdir.as_deref())
.map_err(|err| std::io::Error::other(err.to_string()))?;
}
@@ -1504,6 +1523,18 @@ mod tests {
);
}
#[test]
fn full_enforcement_uses_privileged_setup_and_child_sandbox() {
assert!(ProcessEnforcementMode::Full.uses_privileged_process_setup());
assert!(ProcessEnforcementMode::Full.enforces_child_sandbox());
}
#[test]
fn network_only_enforcement_keeps_child_sandbox_without_privileged_setup() {
assert!(!ProcessEnforcementMode::NetworkOnly.uses_privileged_process_setup());
assert!(ProcessEnforcementMode::NetworkOnly.enforces_child_sandbox());
}
#[cfg(target_os = "linux")]
fn capability_bounding_set_clear_available() -> bool {
capctl::caps::CapState::get_current()
@@ -73,7 +73,7 @@ pub async fn run_process(
// /etc/group so the "sandbox" entry matches. Must run before
// validate_sandbox_user so passwd lookups see the correct identity.
#[cfg(unix)]
if enforcement_mode.enforces_process_controls() {
if enforcement_mode.uses_privileged_process_setup() {
crate::process::update_sandbox_passwd_entries()?;
}
@@ -81,7 +81,7 @@ pub async fn run_process(
// must include a "sandbox" user for privilege dropping; failing fast here
// beats silently running children as root.
#[cfg(unix)]
if enforcement_mode.enforces_process_controls() {
if enforcement_mode.uses_privileged_process_setup() {
crate::process::validate_sandbox_user(policy)?;
crate::process::validate_sandbox_group(policy)?;
}
@@ -90,7 +90,7 @@ pub async fn run_process(
// sandbox user/group. Runs as the supervisor (root) before the child
// is forked so the workload sees writable paths it owns.
#[cfg(unix)]
if enforcement_mode.enforces_process_controls() {
if enforcement_mode.uses_privileged_process_setup() {
crate::process::prepare_filesystem(policy)?;
}
@@ -5,7 +5,7 @@
use landlock::{
ABI, Access, AccessFs, CompatLevel, Compatible, PathBeneath, PathFd, PathFdError, Ruleset,
RulesetAttr, RulesetCreatedAttr,
RulesetAttr,
};
use miette::{IntoDiagnostic, Result};
use openshell_core::policy::{LandlockCompatibility, SandboxPolicy};
@@ -95,6 +95,12 @@ pub struct PreparedRuleset {
compatibility: LandlockCompatibility,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum PathOpenMode {
Privileged,
CurrentUser,
}
/// Phase 1: Open `PathFds` and build the Landlock ruleset **as root**.
///
/// This must run before `drop_privileges()` so that `PathFd::new()` can open
@@ -103,6 +109,27 @@ pub struct PreparedRuleset {
/// Returns `None` if there are no filesystem paths to restrict (no-op).
/// Returns `Some(PreparedRuleset)` on success, or an error.
pub fn prepare(policy: &SandboxPolicy, workdir: Option<&str>) -> Result<Option<PreparedRuleset>> {
prepare_with_path_open_mode(policy, workdir, PathOpenMode::Privileged)
}
/// Phase 1 for already-unprivileged workloads.
///
/// Kubernetes sidecar mode starts the process supervisor as the sandbox UID, so
/// Landlock path FDs are opened as the same UID that will run the workload.
/// Paths this UID cannot open are already unavailable to the workload; omit
/// them from the allowlist and let the resulting ruleset deny everything else.
pub fn prepare_current_user(
policy: &SandboxPolicy,
workdir: Option<&str>,
) -> Result<Option<PreparedRuleset>> {
prepare_with_path_open_mode(policy, workdir, PathOpenMode::CurrentUser)
}
fn prepare_with_path_open_mode(
policy: &SandboxPolicy,
workdir: Option<&str>,
path_open_mode: PathOpenMode,
) -> Result<Option<PreparedRuleset>> {
let read_only = policy.filesystem.read_only.clone();
let mut read_write = policy.filesystem.read_write.clone();
@@ -188,7 +215,7 @@ pub fn prepare(policy: &SandboxPolicy, workdir: Option<&str>) -> Result<Option<P
let mut rules_applied: usize = 0;
for path in &read_only {
if let Some(path_fd) = try_open_path(path, compatibility)? {
if let Some(path_fd) = try_open_path(path, compatibility, path_open_mode)? {
debug!(path = %path.display(), "Landlock allow read-only");
ruleset = ruleset
.add_rule(PathBeneath::new(path_fd, access_read))
@@ -198,7 +225,7 @@ pub fn prepare(policy: &SandboxPolicy, workdir: Option<&str>) -> Result<Option<P
}
for path in &read_write {
if let Some(path_fd) = try_open_path(path, compatibility)? {
if let Some(path_fd) = try_open_path(path, compatibility, path_open_mode)? {
debug!(path = %path.display(), "Landlock allow read-write");
ruleset = ruleset
.add_rule(PathBeneath::new(path_fd, access_all))
@@ -325,7 +352,11 @@ pub fn apply(policy: &SandboxPolicy, workdir: Option<&str>) -> Result<()> {
///
/// In `HardRequirement` mode, any failure is fatal — the caller propagates the
/// error, which ultimately aborts sandbox startup.
fn try_open_path(path: &Path, compatibility: &LandlockCompatibility) -> Result<Option<PathFd>> {
fn try_open_path(
path: &Path,
compatibility: &LandlockCompatibility,
path_open_mode: PathOpenMode,
) -> Result<Option<PathFd>> {
match PathFd::new(path) {
Ok(fd) => Ok(Some(fd)),
Err(err) => {
@@ -335,6 +366,28 @@ fn try_open_path(path: &Path, compatibility: &LandlockCompatibility) -> Result<O
PathFdError::OpenCall { source, .. }
if source.kind() == std::io::ErrorKind::NotFound
);
if matches!(path_open_mode, PathOpenMode::CurrentUser) {
if is_not_found {
debug!(
path = %path.display(),
reason,
"Skipping non-existent Landlock path for current user"
);
} else {
openshell_ocsf::ocsf_emit!(
openshell_ocsf::ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx())
.severity(openshell_ocsf::SeverityId::Informational)
.status(openshell_ocsf::StatusId::Success)
.state(openshell_ocsf::StateId::Other, "already-denied")
.message(format!(
"Skipping inaccessible Landlock path for current user [path:{} error:{err}]",
path.display()
))
.build()
);
}
return Ok(None);
}
match compatibility {
LandlockCompatibility::BestEffort => {
// NotFound is expected for stale baseline paths (e.g.
@@ -421,6 +474,7 @@ mod tests {
let result = try_open_path(
&PathBuf::from("/nonexistent/openshell/test/path"),
&LandlockCompatibility::BestEffort,
PathOpenMode::Privileged,
);
assert!(result.is_ok());
assert!(result.unwrap().is_none());
@@ -431,6 +485,7 @@ mod tests {
let result = try_open_path(
&PathBuf::from("/nonexistent/openshell/test/path"),
&LandlockCompatibility::HardRequirement,
PathOpenMode::Privileged,
);
assert!(result.is_err());
let err_msg = result.unwrap_err().to_string();
@@ -447,11 +502,26 @@ mod tests {
#[test]
fn try_open_path_succeeds_for_existing_path() {
let dir = tempfile::tempdir().unwrap();
let result = try_open_path(dir.path(), &LandlockCompatibility::BestEffort);
let result = try_open_path(
dir.path(),
&LandlockCompatibility::BestEffort,
PathOpenMode::Privileged,
);
assert!(result.is_ok());
assert!(result.unwrap().is_some());
}
#[test]
fn try_open_path_current_user_skips_missing_path_in_hard_requirement() {
let result = try_open_path(
&PathBuf::from("/nonexistent/openshell/test/path"),
&LandlockCompatibility::HardRequirement,
PathOpenMode::CurrentUser,
);
assert!(result.is_ok());
assert!(result.unwrap().is_none());
}
#[test]
fn classify_not_found() {
let err = std::io::Error::from_raw_os_error(libc::ENOENT);
@@ -12,7 +12,7 @@ use std::path::PathBuf;
use std::sync::Once;
/// Opaque handle to a prepared-but-not-yet-enforced sandbox.
/// Holds the Landlock ruleset with `PathFds` opened as root.
/// Holds the Landlock ruleset with `PathFds` opened before child exec.
pub struct PreparedSandbox {
landlock: Option<landlock::PreparedRuleset>,
policy: SandboxPolicy,
@@ -30,6 +30,21 @@ pub fn prepare(policy: &SandboxPolicy, workdir: Option<&str>) -> Result<Prepared
})
}
/// Phase 1 for already-unprivileged workloads.
///
/// Opens Landlock `PathFds` as the current UID. This is used by Kubernetes
/// sidecar mode, where the agent container already runs as the sandbox user.
pub fn prepare_current_user(
policy: &SandboxPolicy,
workdir: Option<&str>,
) -> Result<PreparedSandbox> {
let landlock = landlock::prepare_current_user(policy, workdir)?;
Ok(PreparedSandbox {
landlock,
policy: policy.clone(),
})
}
/// Phase 2: Enforce prepared sandbox restrictions (after `drop_privileges`).
///
/// Calls `restrict_self()` for Landlock and applies seccomp filters.
+15 -25
View File
@@ -65,7 +65,7 @@ fn ssh_server_init(
if let Some(parent) = listen_path.parent() {
std::fs::create_dir_all(parent).into_diagnostic()?;
#[cfg(unix)]
if enforcement_mode.enforces_process_controls() && !shared_socket {
if enforcement_mode.uses_privileged_process_setup() && !shared_socket {
use std::os::unix::fs::PermissionsExt;
let perms = std::fs::Permissions::from_mode(0o700);
std::fs::set_permissions(parent, perms).into_diagnostic()?;
@@ -827,20 +827,15 @@ fn spawn_pty_shell(
// Probe Landlock availability from the parent process where tracing works.
#[cfg(target_os = "linux")]
if enforcement_mode.enforces_process_controls() {
if enforcement_mode.enforces_child_sandbox() {
sandbox::linux::log_sandbox_readiness(policy, workdir.as_deref());
}
// Phase 1 (as root): Prepare Landlock ruleset before drop_privileges.
// Phase 1: Prepare Landlock ruleset before the child applies it.
#[cfg(target_os = "linux")]
let prepared_sandbox = if enforcement_mode.enforces_process_controls() {
Some(
sandbox::linux::prepare(policy, workdir.as_deref())
.map_err(|err| anyhow::anyhow!("Failed to prepare sandbox: {err}"))?,
)
} else {
None
};
let prepared_sandbox =
crate::process::prepare_child_sandbox(policy, workdir.as_deref(), enforcement_mode)
.map_err(|err| anyhow::anyhow!("Failed to prepare sandbox: {err}"))?;
#[cfg(unix)]
{
@@ -986,20 +981,15 @@ fn spawn_pipe_exec(
// Probe Landlock availability from the parent process where tracing works.
#[cfg(target_os = "linux")]
if enforcement_mode.enforces_process_controls() {
if enforcement_mode.enforces_child_sandbox() {
sandbox::linux::log_sandbox_readiness(policy, workdir.as_deref());
}
// Phase 1 (as root): Prepare Landlock ruleset before drop_privileges.
// Phase 1: Prepare Landlock ruleset before the child applies it.
#[cfg(target_os = "linux")]
let prepared_sandbox = if enforcement_mode.enforces_process_controls() {
Some(
sandbox::linux::prepare(policy, workdir.as_deref())
.map_err(|err| anyhow::anyhow!("Failed to prepare sandbox: {err}"))?,
)
} else {
None
};
let prepared_sandbox =
crate::process::prepare_child_sandbox(policy, workdir.as_deref(), enforcement_mode)
.map_err(|err| anyhow::anyhow!("Failed to prepare sandbox: {err}"))?;
#[cfg(unix)]
{
@@ -1157,7 +1147,7 @@ mod unsafe_pty {
#[cfg(target_os = "linux")]
let mut prepared = prepared;
#[cfg(target_os = "linux")]
let supervisor_identity_mount = if enforcement_mode.enforces_process_controls() {
let supervisor_identity_mount = if enforcement_mode.uses_privileged_process_setup() {
crate::process::supervisor_identity_mount_from_env().map_err(|err| {
anyhow::anyhow!("failed to prepare supervisor identity isolation: {err}")
})?
@@ -1205,7 +1195,7 @@ mod unsafe_pty {
#[cfg(target_os = "linux")]
let mut prepared = prepared;
#[cfg(target_os = "linux")]
let supervisor_identity_mount = if enforcement_mode.enforces_process_controls() {
let supervisor_identity_mount = if enforcement_mode.uses_privileged_process_setup() {
crate::process::supervisor_identity_mount_from_env().map_err(|err| {
anyhow::anyhow!("failed to prepare supervisor identity isolation: {err}")
})?
@@ -1260,7 +1250,7 @@ mod unsafe_pty {
// Drop privileges. initgroups/setgid/setuid need /etc/group and
// /etc/passwd which would be blocked if Landlock were already enforced.
if enforcement_mode.enforces_process_controls() {
if enforcement_mode.uses_privileged_process_setup() {
drop_privileges(policy).map_err(|err| std::io::Error::other(err.to_string()))?;
}
crate::process::harden_child_process()
@@ -1275,7 +1265,7 @@ mod unsafe_pty {
}
#[cfg(not(target_os = "linux"))]
if enforcement_mode.enforces_process_controls() {
if enforcement_mode.enforces_child_sandbox() {
sandbox::apply(policy, None).map_err(|err| std::io::Error::other(err.to_string()))?;
}
+1 -1
View File
@@ -237,7 +237,7 @@ add `ci/values-spire.yaml` to the OpenShell release values files.
| supervisor.image.pullPolicy | string | `""` | Supervisor image pull policy. Defaults to the gateway image pull policy when empty. |
| supervisor.image.repository | string | `"ghcr.io/nvidia/openshell/supervisor"` | Supervisor image repository. |
| supervisor.image.tag | string | `""` | Supervisor image tag. Defaults to the chart appVersion when empty. |
| supervisor.proxyUid | int | `1337` | UID for the long-running network sidecar in sidecar topology. The network init container installs nftables rules that exempt this UID. |
| supervisor.sidecar.proxyUid | int | `1337` | UID for the long-running network sidecar in sidecar topology. The network init container installs nftables rules that exempt this UID. |
| supervisor.sideloadMethod | string | `""` | How the supervisor binary is delivered into sandbox pods. Empty (default) = auto-detect from cluster version: K8s >= v1.35 -> "image-volume" (ImageVolume enabled by default; GA in v1.36) K8s < v1.35 -> "init-container" (copies via init container + emptyDir) On K8s v1.33-v1.34 with the ImageVolume feature gate manually enabled, set this to "image-volume" explicitly. |
| supervisor.topology | string | `"combined"` | Supervisor pod topology for Kubernetes sandboxes. "combined" runs the current single supervisor container in the agent pod. "sidecar" runs network enforcement in a dedicated sidecar and the process supervisor as a low-capability wrapper in the agent container. |
| tolerations | list | `[]` | Tolerations for the gateway pod. |
@@ -113,8 +113,7 @@ data:
grpc_endpoint = {{ include "openshell.grpcEndpoint" . | quote }}
service_account_name = {{ include "openshell.sandboxServiceAccountName" . | quote }}
supervisor_sideload_method = {{ include "openshell.supervisorSideloadMethod" . | quote }}
supervisor_topology = {{ .Values.supervisor.topology | default "combined" | quote }}
proxy_uid = {{ .Values.supervisor.proxyUid | default 1337 }}
topology = {{ .Values.supervisor.topology | default "combined" | quote }}
sa_token_ttl_secs = {{ .Values.server.sandboxJwt.k8sSaTokenTtlSecs | default 3600 }}
{{- if .Values.server.providerTokenGrants.spiffe.enabled }}
provider_spiffe_workload_api_socket_path = {{ .Values.server.providerTokenGrants.spiffe.workloadApiSocketPath | quote }}
@@ -143,3 +142,6 @@ data:
{{- if .Values.supervisor.image.pullPolicy }}
supervisor_image_pull_policy = {{ .Values.supervisor.image.pullPolicy | quote }}
{{- end }}
[openshell.drivers.kubernetes.sidecar]
proxy_uid = {{ .Values.supervisor.sidecar.proxyUid | default 1337 }}
@@ -83,23 +83,26 @@ tests:
path: data["gateway.toml"]
pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?service_account_name\s*=\s*"openshell-sandbox"'
- it: renders supervisor topology under [openshell.drivers.kubernetes]
- it: renders topology under [openshell.drivers.kubernetes]
template: templates/gateway-config.yaml
set:
supervisor.topology: sidecar
asserts:
- matchRegex:
path: data["gateway.toml"]
pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?supervisor_topology\s*=\s*"sidecar"'
pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?topology\s*=\s*"sidecar"'
- notMatchRegex:
path: data["gateway.toml"]
pattern: 'supervisor_topology\s*='
- it: renders proxy uid under [openshell.drivers.kubernetes]
- it: renders proxy uid under [openshell.drivers.kubernetes.sidecar]
template: templates/gateway-config.yaml
set:
supervisor.proxyUid: 2200
supervisor.sidecar.proxyUid: 2200
asserts:
- matchRegex:
path: data["gateway.toml"]
pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?proxy_uid\s*=\s*2200'
pattern: '(?ms)\[openshell\.drivers\.kubernetes\.sidecar\].*?proxy_uid\s*=\s*2200'
- it: renders sandbox image pull secrets under [openshell.drivers.kubernetes]
template: templates/gateway-config.yaml
+4 -3
View File
@@ -49,9 +49,10 @@ supervisor:
# "sidecar" runs network enforcement in a dedicated sidecar and the process
# supervisor as a low-capability wrapper in the agent container.
topology: "combined"
# -- UID for the long-running network sidecar in sidecar topology. The
# network init container installs nftables rules that exempt this UID.
proxyUid: 1337
sidecar:
# -- UID for the long-running network sidecar in sidecar topology. The
# network init container installs nftables rules that exempt this UID.
proxyUid: 1337
# -- Image pull secrets attached to gateway and helper pods.
imagePullSecrets: []
+1 -1
View File
@@ -161,7 +161,7 @@ The most commonly changed values are:
| `pkiInitJob.serverDnsNames` / `certManager.serverDnsNames` | Additional gateway server DNS SANs. Wildcard SANs also enable sandbox service URLs under that domain. |
| `supervisor.sideloadMethod` | How the supervisor binary is delivered into sandbox pods. Leave empty to auto-detect based on cluster version: clusters running Kubernetes 1.35 or later use `image-volume` (ImageVolume GA in 1.36); older clusters use `init-container`. Set explicitly to `image-volume` on Kubernetes 1.33 or 1.34 with the ImageVolume feature gate enabled, or to `init-container` to force the legacy path on any version. |
| `supervisor.topology` | Sandbox pod topology. Refer to [Topology](/kubernetes/topology). |
| `supervisor.proxyUid` | Non-root UID for the long-running network sidecar when `supervisor.topology=sidecar`. The UID must not match the sandbox UID. |
| `supervisor.sidecar.proxyUid` | Non-root UID for the long-running network sidecar when `supervisor.topology=sidecar`. The UID must not match the sandbox UID. |
Use a values file for repeatable deployments:
+24 -16
View File
@@ -21,7 +21,7 @@ lower-privilege agent container.
| Topology | Use when | Main tradeoff |
|---|---|---|
| `combined` | You need OpenShell network, filesystem, and process controls in the sandbox workload. | The agent container carries the Linux capabilities the supervisor needs. |
| `sidecar` | You need the agent container to run as non-root without added Linux capabilities, and network policy is the primary control. | The process supervisor is network-only, so filesystem/process controls do not run in the agent container. |
| `sidecar` | You need the agent container to run as non-root without added Linux capabilities, and network policy is the primary control. | Privilege-dropping and supervisor mount isolation do not run in the agent container. |
## Privilege Model
@@ -121,7 +121,7 @@ The pod contains these OpenShell-managed pieces:
| Component | Runs as | Purpose |
|---|---|---|
| Network init container | Root with setup capabilities | Installs pod-level nftables rules and prepares shared sidecar state. |
| Network sidecar | `supervisor.proxyUid` | Runs the proxy, enforces network policy, owns gateway authentication and the gateway session, and writes proxy TLS plus local policy/provider snapshots. |
| Network sidecar | `supervisor.sidecar.proxyUid` | Runs the proxy, enforces network policy, owns gateway authentication and the gateway session, and writes proxy TLS plus local policy/provider snapshots. |
| Agent container | Resolved sandbox UID/GID | Runs the process supervisor and launches the user workload. |
In this topology, the agent container defaults to `runAsNonRoot: true`,
@@ -137,10 +137,14 @@ default sidecar path.
<Warning>
Sidecar mode runs the process supervisor in `network-only` mode. OpenShell still
enforces network endpoint and L7 policy through the sidecar, but the process
supervisor does not apply Landlock filesystem policy, process privilege
dropping, or process/binary identity checks. Use `combined` topology when you
need those controls in the same supervisor path.
enforces network endpoint and L7 policy through the sidecar, and the process
supervisor applies Landlock filesystem policy and child seccomp filters where
the kernel/runtime supports them. The process supervisor does not perform
root-to-sandbox privilege dropping because Kubernetes starts the container as
the sandbox UID/GID, and it does not perform supervisor identity mount
isolation because gateway credentials are not mounted into the agent container.
Sidecar pods use `shareProcessNamespace: true` so the network sidecar can
resolve workload process and binary identity through `/proc/<entrypoint-pid>`.
</Warning>
## Credential Exposure
@@ -161,9 +165,9 @@ watches for newer revisions that the network sidecar writes after settings
polls, so future child processes can see refreshed provider env without giving
the agent container gateway authentication material. This does not mutate the
environment of the already-running workload entrypoint. Use `combined` topology
when you need full process/filesystem enforcement in the same supervisor path;
use additional runtime isolation when you need a stronger container boundary
around network-only sidecar workloads.
when you need the full single-supervisor enforcement path; use additional
runtime isolation when you need a stronger container boundary around sidecar
workloads.
## RuntimeClass Isolation
@@ -172,9 +176,10 @@ Containers when the cluster supports them. A sandboxed runtime strengthens the
container boundary while OpenShell focuses on network policy enforcement from
the sidecar.
Runtime classes do not re-enable the OpenShell filesystem and process controls
that sidecar mode relaxes. Use them as an additional workload boundary, not as a
replacement for the combined topology's full supervisor controls.
Runtime classes do not re-enable the OpenShell privilege-drop or supervisor
mount-isolation controls that sidecar mode relaxes. Use them as an additional
workload boundary, not as a replacement for the combined topology's full
supervisor controls.
You can set a default runtime class in the Kubernetes driver configuration or
override it per sandbox with driver config:
@@ -191,7 +196,9 @@ For direct gateway TOML configuration, set the Kubernetes driver fields:
```toml
[openshell.drivers.kubernetes]
supervisor_topology = "sidecar"
topology = "sidecar"
[openshell.drivers.kubernetes.sidecar]
proxy_uid = 1337
```
@@ -204,11 +211,12 @@ When the Helm chart renders `gateway.toml`, set the equivalent chart values:
```yaml
supervisor:
topology: sidecar
proxyUid: 1337
sidecar:
proxyUid: 1337
```
Leave `supervisor_topology` unset, or set it to `combined`, to keep the
original single-container supervisor path. For Helm installs, leave
Leave `topology` unset, or set it to `combined`, to keep the original
single-container supervisor path. For Helm installs, leave
`supervisor.topology` unset or set it to `combined`.
## Next Steps
+6 -4
View File
@@ -180,10 +180,7 @@ supervisor_sideload_method = "image-volume"
# "combined" runs the existing single supervisor container with full process,
# filesystem, and network enforcement in the agent container. "sidecar" moves
# pod-level network enforcement and gateway session handling into a network sidecar.
supervisor_topology = "combined"
# UID used by the long-running network sidecar. In sidecar topology the
# network init container installs nftables rules that exempt this UID.
proxy_uid = 1337
topology = "combined"
grpc_endpoint = "https://openshell-gateway.agents.svc:8080"
ssh_socket_path = "/run/openshell/ssh.sock"
client_tls_secret_name = "openshell-client-tls"
@@ -207,6 +204,11 @@ provider_spiffe_workload_api_socket_path = "/spiffe-workload-api/spire-agent.soc
# back to 1000 on non-OpenShift clusters.
# sandbox_uid = 1500
# sandbox_gid = 1500
[openshell.drivers.kubernetes.sidecar]
# UID used by the long-running network sidecar. In sidecar topology the
# network init container installs nftables rules that exempt this UID.
proxy_uid = 1337
```
### Docker
+8 -6
View File
@@ -306,8 +306,8 @@ For maintainer-level implementation details, refer to the [Kubernetes driver REA
| `supervisor_image` | `supervisor.image.repository` / `supervisor.image.tag` | Set the supervisor image that provides the `openshell-sandbox` binary. |
| `supervisor_image_pull_policy` | `supervisor.image.pullPolicy` | Set the Kubernetes image pull policy for the supervisor image. |
| `supervisor_sideload_method` | `supervisor.sideloadMethod` | How the supervisor binary is delivered into sandbox pods. Leave empty to auto-detect from cluster version. Set to `image-volume` to mount the supervisor OCI image directly as a volume (requires Kubernetes 1.33+ with the ImageVolume feature gate; GA in 1.36), or `init-container` to copy it through an init container on older clusters. |
| `supervisor_topology` | `supervisor.topology` | Set `combined` for the default single supervisor path, or `sidecar` to move pod-level network enforcement and the gateway session into a dedicated sidecar. |
| `proxy_uid` | `supervisor.proxyUid` | UID used by the long-running network sidecar in `sidecar` topology. The network init container exempts this UID from proxy redirection. |
| `topology` | `supervisor.topology` | Set `combined` for the default single supervisor path, or `sidecar` to move pod-level network enforcement and the gateway session into a dedicated sidecar. |
| `sidecar.proxy_uid` | `supervisor.sidecar.proxyUid` | UID used by the long-running network sidecar in `sidecar` topology. The network init container exempts this UID from proxy redirection. |
| `app_armor_profile` | `server.appArmorProfile` | Set the sandbox agent container's AppArmor profile. Helm defaults this to `Unconfined` so AppArmor-enabled nodes do not block supervisor network namespace setup. Set the Helm value to an empty string to omit the field, or use `RuntimeDefault` or `Localhost/<profile-name>` for operator-managed profiles. |
| `workspace_default_storage_size` | `server.workspaceDefaultStorageSize` | Set the default workspace PVC size for new sandboxes. |
| `sa_token_ttl_secs` | `server.sandboxJwt.k8sSaTokenTtlSecs` | Set the projected ServiceAccount token TTL used for the bootstrap token exchange. |
@@ -324,10 +324,12 @@ sandbox bootstrap token or client TLS secret in the default sidecar path. The
provider environment snapshot is refreshed by the network sidecar after
settings polls so future child processes can see updated provider env without
gateway access in the agent container.
Sidecar mode keeps gateway session and SSH behavior, but the process supervisor
runs in `network-only` mode: filesystem policy, process privilege dropping, and
process/binary identity checks are not applied by the process container. Network
policy still runs in the sidecar and remains endpoint/L7 scoped.
Sidecar mode keeps gateway session and SSH behavior. The process supervisor
applies Landlock filesystem policy and child seccomp filters where supported,
but it does not perform root-to-sandbox privilege dropping or supervisor
identity mount isolation. Network policy still runs in the sidecar, and sidecar
pods set `shareProcessNamespace: true` so the network sidecar can resolve
process/binary identity through `/proc/<entrypoint-pid>`.
The Kubernetes driver creates namespaced `agents.x-k8s.io` `Sandbox` resources from the Kubernetes SIG Apps [agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) project. It detects the served Sandbox API at runtime, caches the selected API version for the gateway process, and uses `v1beta1` when available before falling back to `v1alpha1`, so supported Agent Sandbox installations work without version-specific operator configuration. The Agent Sandbox controller turns those resources into sandbox pods and related storage.