mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-30 07:33:59 +08:00
fix(kubernetes): preserve process identity in sidecar topology
Render sidecar pods with a shared process namespace, keep binary-aware network policy enabled, and move Kubernetes sidecar settings under the nested sidecar config table. Also apply unprivileged Landlock/seccomp setup in NetworkOnly supervisor mode so sidecar topology keeps sandbox child hardening without privileged process setup. Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
This commit is contained in:
@@ -107,8 +107,12 @@ log expressions can fail without rolling back the required table, chain, and
|
||||
reject rules.
|
||||
The agent container runs as the resolved sandbox UID/GID with no added Linux
|
||||
capabilities. Sidecar mode preserves gateway session and SSH behavior, but
|
||||
treats the process leaf as network-only: Landlock filesystem policy, process
|
||||
privilege dropping, and process/binary identity checks are not applied there.
|
||||
treats the process leaf as network-only: Landlock filesystem policy and child
|
||||
seccomp still apply where supported, while process privilege dropping and
|
||||
supervisor identity mount isolation do not run because the agent container is
|
||||
already unprivileged. Sidecar pods use a shared process namespace so the
|
||||
network sidecar can resolve workload process and binary identity through
|
||||
`/proc/<entrypoint-pid>`.
|
||||
|
||||
## Images
|
||||
|
||||
|
||||
@@ -62,15 +62,17 @@ container and runs the long-lived network sidecar as a non-root UID with no
|
||||
added Linux capabilities. The agent container also runs as the resolved sandbox
|
||||
UID/GID with `allowPrivilegeEscalation: false` and `capabilities.drop: ["ALL"]`.
|
||||
In this mode OpenShell preserves gateway session and SSH behavior, but the
|
||||
process supervisor runs in network-only mode and does not apply Landlock
|
||||
filesystem policy, process privilege dropping, or process/binary identity
|
||||
checks. Network endpoint and L7 policy remain enforced by the network sidecar.
|
||||
process supervisor does not perform root-to-sandbox privilege dropping or
|
||||
supervisor identity mount isolation. It still applies Landlock filesystem policy
|
||||
and child seccomp filters where the kernel/runtime supports them. Network
|
||||
endpoint and L7 policy remain enforced by the network sidecar, and
|
||||
sidecar pods use a shared process namespace so the network sidecar can resolve
|
||||
process/binary identity through `/proc/<entrypoint-pid>`.
|
||||
|
||||
Sidecar mode uses the pod `fsGroup` to make the projected service-account token
|
||||
and sandbox client TLS secret group-readable so the non-root process supervisor
|
||||
can authenticate to the gateway. Treat the agent container as trusted with
|
||||
respect to those in-pod gateway credentials until a narrower credential handoff
|
||||
exists.
|
||||
Sidecar mode keeps gateway credentials in the network sidecar. The agent
|
||||
container does not mount the projected service-account token used for sandbox
|
||||
token bootstrap, does not mount the sandbox client TLS secret, and does not get
|
||||
gateway callback environment variables.
|
||||
|
||||
The driver can request a Kubernetes AppArmor profile through
|
||||
`app_armor_profile`.
|
||||
|
||||
@@ -88,6 +88,35 @@ impl FromStr for SupervisorTopology {
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(default, deny_unknown_fields)]
|
||||
pub struct KubernetesSidecarConfig {
|
||||
/// UID used by the long-running network sidecar in `sidecar` topology.
|
||||
/// The network init container installs nftables rules that exempt this
|
||||
/// UID, so it must not match the sandbox workload UID.
|
||||
pub proxy_uid: u32,
|
||||
}
|
||||
|
||||
impl Default for KubernetesSidecarConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
proxy_uid: DEFAULT_PROXY_UID,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl KubernetesSidecarConfig {
|
||||
pub fn validate_proxy_uid(&self) -> Result<(), String> {
|
||||
if self.proxy_uid < openshell_policy::MIN_SANDBOX_UID {
|
||||
return Err(format!(
|
||||
"sidecar.proxy_uid must be at least {}",
|
||||
openshell_policy::MIN_SANDBOX_UID
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// Kubernetes `AppArmor` profile requested for the sandbox agent container.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum AppArmorProfile {
|
||||
@@ -213,11 +242,9 @@ pub struct KubernetesComputeConfig {
|
||||
/// How the supervisor binary is delivered into sandbox pods.
|
||||
pub supervisor_sideload_method: SupervisorSideloadMethod,
|
||||
/// How the supervisor is arranged for Kubernetes sandbox pods.
|
||||
pub supervisor_topology: SupervisorTopology,
|
||||
/// UID used by the long-running network sidecar in `sidecar` topology.
|
||||
/// The network init container installs nftables rules that exempt this
|
||||
/// UID, so it must not match the sandbox workload UID.
|
||||
pub proxy_uid: u32,
|
||||
pub topology: SupervisorTopology,
|
||||
/// Sidecar-only settings used when `topology = "sidecar"`.
|
||||
pub sidecar: KubernetesSidecarConfig,
|
||||
pub grpc_endpoint: String,
|
||||
pub ssh_socket_path: String,
|
||||
pub client_tls_secret_name: String,
|
||||
@@ -303,8 +330,8 @@ impl Default for KubernetesComputeConfig {
|
||||
supervisor_image: DEFAULT_SUPERVISOR_IMAGE.to_string(),
|
||||
supervisor_image_pull_policy: String::new(),
|
||||
supervisor_sideload_method: SupervisorSideloadMethod::default(),
|
||||
supervisor_topology: SupervisorTopology::default(),
|
||||
proxy_uid: DEFAULT_PROXY_UID,
|
||||
topology: SupervisorTopology::default(),
|
||||
sidecar: KubernetesSidecarConfig::default(),
|
||||
grpc_endpoint: String::new(),
|
||||
ssh_socket_path: "/run/openshell/ssh.sock".to_string(),
|
||||
client_tls_secret_name: String::new(),
|
||||
@@ -350,13 +377,7 @@ impl KubernetesComputeConfig {
|
||||
}
|
||||
|
||||
pub fn validate_proxy_uid(&self) -> Result<(), String> {
|
||||
if self.proxy_uid < openshell_policy::MIN_SANDBOX_UID {
|
||||
return Err(format!(
|
||||
"proxy_uid must be at least {}",
|
||||
openshell_policy::MIN_SANDBOX_UID
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
self.sidecar.validate_proxy_uid()
|
||||
}
|
||||
|
||||
/// Resolve the sandbox UID/GID pair.
|
||||
@@ -486,50 +507,63 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn default_supervisor_topology_is_combined() {
|
||||
fn default_topology_is_combined() {
|
||||
let cfg = KubernetesComputeConfig::default();
|
||||
assert_eq!(cfg.supervisor_topology, SupervisorTopology::Combined);
|
||||
assert_eq!(cfg.supervisor_topology.to_string(), "combined");
|
||||
assert_eq!(cfg.topology, SupervisorTopology::Combined);
|
||||
assert_eq!(cfg.topology.to_string(), "combined");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn default_proxy_uid_is_dedicated_non_root_uid() {
|
||||
let cfg = KubernetesComputeConfig::default();
|
||||
assert_eq!(cfg.proxy_uid, DEFAULT_PROXY_UID);
|
||||
assert_eq!(cfg.sidecar.proxy_uid, DEFAULT_PROXY_UID);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn serde_override_supervisor_topology_sidecar() {
|
||||
fn serde_override_topology_sidecar() {
|
||||
let json = serde_json::json!({
|
||||
"supervisor_topology": "sidecar"
|
||||
"topology": "sidecar"
|
||||
});
|
||||
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
|
||||
assert_eq!(cfg.supervisor_topology, SupervisorTopology::Sidecar);
|
||||
assert_eq!(cfg.topology, SupervisorTopology::Sidecar);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn serde_override_supervisor_topology_combined() {
|
||||
fn serde_override_topology_combined() {
|
||||
let json = serde_json::json!({
|
||||
"supervisor_topology": "combined"
|
||||
"topology": "combined"
|
||||
});
|
||||
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
|
||||
assert_eq!(cfg.supervisor_topology, SupervisorTopology::Combined);
|
||||
assert_eq!(cfg.topology, SupervisorTopology::Combined);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn serde_override_proxy_uid() {
|
||||
fn serde_rejects_sidecar_binary_identity_field() {
|
||||
let json = serde_json::json!({
|
||||
"proxy_uid": 2000
|
||||
"sidecar": {
|
||||
"binary_identity": "shared-pid"
|
||||
}
|
||||
});
|
||||
let err = serde_json::from_value::<KubernetesComputeConfig>(json).unwrap_err();
|
||||
assert!(err.to_string().contains("unknown field"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn serde_override_sidecar_proxy_uid_nested() {
|
||||
let json = serde_json::json!({
|
||||
"sidecar": {
|
||||
"proxy_uid": 2000
|
||||
}
|
||||
});
|
||||
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
|
||||
assert_eq!(cfg.proxy_uid, 2000);
|
||||
assert_eq!(cfg.sidecar.proxy_uid, 2000);
|
||||
cfg.validate_proxy_uid().unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn validate_proxy_uid_rejects_privileged_uid() {
|
||||
let cfg = KubernetesComputeConfig {
|
||||
proxy_uid: 999,
|
||||
sidecar: KubernetesSidecarConfig { proxy_uid: 999 },
|
||||
..KubernetesComputeConfig::default()
|
||||
};
|
||||
let err = cfg.validate_proxy_uid().unwrap_err();
|
||||
@@ -537,14 +571,34 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn serde_rejects_invalid_supervisor_topology() {
|
||||
fn serde_rejects_invalid_topology() {
|
||||
let json = serde_json::json!({
|
||||
"supervisor_topology": "unsupported"
|
||||
"topology": "unsupported"
|
||||
});
|
||||
let err = serde_json::from_value::<KubernetesComputeConfig>(json).unwrap_err();
|
||||
assert!(err.to_string().contains("unknown variant"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn serde_rejects_removed_supervisor_topology_field() {
|
||||
let json = serde_json::json!({
|
||||
"supervisor_topology": "sidecar"
|
||||
});
|
||||
let err = serde_json::from_value::<KubernetesComputeConfig>(json).unwrap_err();
|
||||
assert!(err.to_string().contains("unknown field"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn serde_rejects_removed_flat_sidecar_fields() {
|
||||
for json in [
|
||||
serde_json::json!({ "sidecar_binary_identity": "shared-pid" }),
|
||||
serde_json::json!({ "proxy_uid": 2000 }),
|
||||
] {
|
||||
let err = serde_json::from_value::<KubernetesComputeConfig>(json).unwrap_err();
|
||||
assert!(err.to_string().contains("unknown field"));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn serde_rejects_removed_process_enforcement_field() {
|
||||
let json = serde_json::json!({
|
||||
|
||||
@@ -563,8 +563,8 @@ impl KubernetesComputeDriver {
|
||||
supervisor_image: &self.config.supervisor_image,
|
||||
supervisor_image_pull_policy: &self.config.supervisor_image_pull_policy,
|
||||
supervisor_sideload_method: self.config.supervisor_sideload_method,
|
||||
supervisor_topology: self.config.supervisor_topology,
|
||||
proxy_uid: self.config.proxy_uid,
|
||||
supervisor_topology: self.config.topology,
|
||||
proxy_uid: self.config.sidecar.proxy_uid,
|
||||
service_account_name: &self.config.service_account_name,
|
||||
sandbox_id: &sandbox.id,
|
||||
sandbox_name: &sandbox.name,
|
||||
@@ -1334,11 +1334,6 @@ fn supervisor_sidecar_env(
|
||||
openshell_core::sandbox_env::NETWORK_ENFORCEMENT_MODE,
|
||||
"sidecar-nftables",
|
||||
);
|
||||
upsert_env(
|
||||
&mut env,
|
||||
openshell_core::sandbox_env::NETWORK_BINARY_IDENTITY,
|
||||
"relaxed",
|
||||
);
|
||||
upsert_env(
|
||||
&mut env,
|
||||
openshell_core::sandbox_env::SUPERVISOR_READY_FILE,
|
||||
@@ -1488,6 +1483,8 @@ fn apply_supervisor_sidecar_topology(
|
||||
sc.insert("fsGroup".to_string(), serde_json::json!(params.sandbox_gid));
|
||||
}
|
||||
|
||||
spec.insert("shareProcessNamespace".to_string(), serde_json::json!(true));
|
||||
|
||||
apply_supervisor_binary_source(
|
||||
spec,
|
||||
params.supervisor_image,
|
||||
@@ -3124,10 +3121,7 @@ mod tests {
|
||||
¶ms,
|
||||
);
|
||||
|
||||
assert!(
|
||||
pod_template["spec"]["shareProcessNamespace"].is_null(),
|
||||
"sidecar mode no longer needs a shared process namespace when binary identity is relaxed"
|
||||
);
|
||||
assert_eq!(pod_template["spec"]["shareProcessNamespace"], true);
|
||||
assert_eq!(pod_template["spec"]["securityContext"]["fsGroup"], 1500);
|
||||
let containers = pod_template["spec"]["containers"].as_array().unwrap();
|
||||
assert_eq!(containers.len(), 2);
|
||||
@@ -3250,7 +3244,7 @@ mod tests {
|
||||
sidecar,
|
||||
openshell_core::sandbox_env::NETWORK_BINARY_IDENTITY
|
||||
),
|
||||
Some("relaxed")
|
||||
None
|
||||
);
|
||||
assert_eq!(
|
||||
rendered_env(sidecar, openshell_core::sandbox_env::ENTRYPOINT_PID_FILE),
|
||||
|
||||
@@ -7,8 +7,8 @@ pub mod grpc;
|
||||
|
||||
pub use config::{
|
||||
AppArmorProfile, DEFAULT_PROXY_UID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME,
|
||||
DEFAULT_WORKSPACE_STORAGE_SIZE, KubernetesComputeConfig, SupervisorSideloadMethod,
|
||||
SupervisorTopology,
|
||||
DEFAULT_WORKSPACE_STORAGE_SIZE, KubernetesComputeConfig, KubernetesSidecarConfig,
|
||||
SupervisorSideloadMethod, SupervisorTopology,
|
||||
};
|
||||
pub use driver::{KubernetesComputeDriver, KubernetesDriverError};
|
||||
pub use grpc::ComputeDriverService;
|
||||
|
||||
@@ -11,7 +11,8 @@ use openshell_core::VERSION;
|
||||
use openshell_core::proto::compute::v1::compute_driver_server::ComputeDriverServer;
|
||||
use openshell_driver_kubernetes::{
|
||||
AppArmorProfile, ComputeDriverService, DEFAULT_PROXY_UID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME,
|
||||
KubernetesComputeConfig, KubernetesComputeDriver, SupervisorSideloadMethod, SupervisorTopology,
|
||||
KubernetesComputeConfig, KubernetesComputeDriver, KubernetesSidecarConfig,
|
||||
SupervisorSideloadMethod, SupervisorTopology,
|
||||
};
|
||||
|
||||
#[derive(Parser, Debug)]
|
||||
@@ -82,13 +83,19 @@ struct Args {
|
||||
|
||||
#[arg(
|
||||
long,
|
||||
env = "OPENSHELL_SUPERVISOR_TOPOLOGY",
|
||||
alias = "supervisor-topology",
|
||||
env = "OPENSHELL_K8S_TOPOLOGY",
|
||||
default_value = "combined"
|
||||
)]
|
||||
supervisor_topology: SupervisorTopology,
|
||||
topology: SupervisorTopology,
|
||||
|
||||
#[arg(long, env = "OPENSHELL_PROXY_UID", default_value_t = DEFAULT_PROXY_UID)]
|
||||
proxy_uid: u32,
|
||||
#[arg(
|
||||
long = "sidecar-proxy-uid",
|
||||
alias = "proxy-uid",
|
||||
env = "OPENSHELL_K8S_SIDECAR_PROXY_UID",
|
||||
default_value_t = DEFAULT_PROXY_UID
|
||||
)]
|
||||
sidecar_proxy_uid: u32,
|
||||
|
||||
#[arg(long, env = "OPENSHELL_ENABLE_USER_NAMESPACES")]
|
||||
enable_user_namespaces: bool,
|
||||
@@ -133,8 +140,10 @@ async fn main() -> Result<()> {
|
||||
.unwrap_or_else(|| openshell_core::config::DEFAULT_SUPERVISOR_IMAGE.to_string()),
|
||||
supervisor_image_pull_policy: args.supervisor_image_pull_policy.unwrap_or_default(),
|
||||
supervisor_sideload_method: args.supervisor_sideload_method,
|
||||
supervisor_topology: args.supervisor_topology,
|
||||
proxy_uid: args.proxy_uid,
|
||||
topology: args.topology,
|
||||
sidecar: KubernetesSidecarConfig {
|
||||
proxy_uid: args.sidecar_proxy_uid,
|
||||
},
|
||||
grpc_endpoint: args.grpc_endpoint.unwrap_or_default(),
|
||||
ssh_socket_path: args.sandbox_ssh_socket_path,
|
||||
client_tls_secret_name: args.client_tls_secret_name.unwrap_or_default(),
|
||||
|
||||
@@ -42,9 +42,32 @@ pub enum ProcessEnforcementMode {
|
||||
|
||||
impl ProcessEnforcementMode {
|
||||
#[must_use]
|
||||
pub const fn enforces_process_controls(self) -> bool {
|
||||
pub const fn uses_privileged_process_setup(self) -> bool {
|
||||
matches!(self, Self::Full)
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub const fn enforces_child_sandbox(self) -> bool {
|
||||
matches!(self, Self::Full | Self::NetworkOnly)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
pub(crate) fn prepare_child_sandbox(
|
||||
policy: &SandboxPolicy,
|
||||
workdir: Option<&str>,
|
||||
enforcement_mode: ProcessEnforcementMode,
|
||||
) -> Result<Option<sandbox::linux::PreparedSandbox>> {
|
||||
if !enforcement_mode.enforces_child_sandbox() {
|
||||
return Ok(None);
|
||||
}
|
||||
|
||||
let prepared = if enforcement_mode.uses_privileged_process_setup() {
|
||||
sandbox::linux::prepare(policy, workdir)
|
||||
} else {
|
||||
sandbox::linux::prepare_current_user(policy, workdir)
|
||||
}?;
|
||||
Ok(Some(prepared))
|
||||
}
|
||||
|
||||
const SUPERVISOR_ONLY_ENV_VARS: &[&str] = &[
|
||||
@@ -580,24 +603,20 @@ impl ProcessHandle {
|
||||
// process where the tracing subscriber is functional. The child's
|
||||
// pre_exec context cannot reliably emit structured logs.
|
||||
#[cfg(target_os = "linux")]
|
||||
if enforcement_mode.enforces_process_controls() {
|
||||
if enforcement_mode.enforces_child_sandbox() {
|
||||
sandbox::linux::log_sandbox_readiness(policy, workdir);
|
||||
}
|
||||
|
||||
// Phase 1 (as root): Prepare Landlock ruleset by opening PathFds.
|
||||
// This MUST happen before drop_privileges() so that root-only paths
|
||||
// (e.g. mode 700 directories) can be opened. See issue #803.
|
||||
// Phase 1: Prepare Landlock ruleset by opening PathFds.
|
||||
// In full mode this runs before drop_privileges() so root-only paths
|
||||
// can be opened. In sidecar network-only mode the container already
|
||||
// runs as the sandbox UID, so inaccessible paths are unavailable to
|
||||
// the workload and best-effort compatibility skips them.
|
||||
#[cfg(target_os = "linux")]
|
||||
let prepared_sandbox = if enforcement_mode.enforces_process_controls() {
|
||||
Some(
|
||||
sandbox::linux::prepare(policy, workdir)
|
||||
.map_err(|err| miette::miette!("Failed to prepare sandbox: {err}"))?,
|
||||
)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let prepared_sandbox = prepare_child_sandbox(policy, workdir, enforcement_mode)
|
||||
.map_err(|err| miette::miette!("Failed to prepare sandbox: {err}"))?;
|
||||
#[cfg(target_os = "linux")]
|
||||
let supervisor_identity_mount = if enforcement_mode.enforces_process_controls() {
|
||||
let supervisor_identity_mount = if enforcement_mode.uses_privileged_process_setup() {
|
||||
supervisor_identity_mount_from_env().map_err(|err| {
|
||||
miette::miette!("Failed to prepare supervisor identity isolation: {err}")
|
||||
})?
|
||||
@@ -640,7 +659,7 @@ impl ProcessHandle {
|
||||
// Drop privileges. initgroups/setgid/setuid need access to
|
||||
// /etc/group and /etc/passwd which would be blocked if
|
||||
// Landlock were already enforced.
|
||||
if enforcement_mode.enforces_process_controls() {
|
||||
if enforcement_mode.uses_privileged_process_setup() {
|
||||
drop_privileges(&policy)
|
||||
.map_err(|err| std::io::Error::other(err.to_string()))?;
|
||||
}
|
||||
@@ -741,14 +760,14 @@ impl ProcessHandle {
|
||||
// Drop privileges before applying sandbox restrictions.
|
||||
// initgroups/setgid/setuid need access to /etc/group and /etc/passwd
|
||||
// which may be blocked by Landlock.
|
||||
if enforcement_mode.enforces_process_controls() {
|
||||
if enforcement_mode.uses_privileged_process_setup() {
|
||||
drop_privileges(&policy)
|
||||
.map_err(|err| std::io::Error::other(err.to_string()))?;
|
||||
}
|
||||
|
||||
harden_child_process().map_err(|err| std::io::Error::other(err.to_string()))?;
|
||||
|
||||
if enforcement_mode.enforces_process_controls() {
|
||||
if enforcement_mode.enforces_child_sandbox() {
|
||||
sandbox::apply(&policy, workdir.as_deref())
|
||||
.map_err(|err| std::io::Error::other(err.to_string()))?;
|
||||
}
|
||||
@@ -1504,6 +1523,18 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn full_enforcement_uses_privileged_setup_and_child_sandbox() {
|
||||
assert!(ProcessEnforcementMode::Full.uses_privileged_process_setup());
|
||||
assert!(ProcessEnforcementMode::Full.enforces_child_sandbox());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn network_only_enforcement_keeps_child_sandbox_without_privileged_setup() {
|
||||
assert!(!ProcessEnforcementMode::NetworkOnly.uses_privileged_process_setup());
|
||||
assert!(ProcessEnforcementMode::NetworkOnly.enforces_child_sandbox());
|
||||
}
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
fn capability_bounding_set_clear_available() -> bool {
|
||||
capctl::caps::CapState::get_current()
|
||||
|
||||
@@ -73,7 +73,7 @@ pub async fn run_process(
|
||||
// /etc/group so the "sandbox" entry matches. Must run before
|
||||
// validate_sandbox_user so passwd lookups see the correct identity.
|
||||
#[cfg(unix)]
|
||||
if enforcement_mode.enforces_process_controls() {
|
||||
if enforcement_mode.uses_privileged_process_setup() {
|
||||
crate::process::update_sandbox_passwd_entries()?;
|
||||
}
|
||||
|
||||
@@ -81,7 +81,7 @@ pub async fn run_process(
|
||||
// must include a "sandbox" user for privilege dropping; failing fast here
|
||||
// beats silently running children as root.
|
||||
#[cfg(unix)]
|
||||
if enforcement_mode.enforces_process_controls() {
|
||||
if enforcement_mode.uses_privileged_process_setup() {
|
||||
crate::process::validate_sandbox_user(policy)?;
|
||||
crate::process::validate_sandbox_group(policy)?;
|
||||
}
|
||||
@@ -90,7 +90,7 @@ pub async fn run_process(
|
||||
// sandbox user/group. Runs as the supervisor (root) before the child
|
||||
// is forked so the workload sees writable paths it owns.
|
||||
#[cfg(unix)]
|
||||
if enforcement_mode.enforces_process_controls() {
|
||||
if enforcement_mode.uses_privileged_process_setup() {
|
||||
crate::process::prepare_filesystem(policy)?;
|
||||
}
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
|
||||
use landlock::{
|
||||
ABI, Access, AccessFs, CompatLevel, Compatible, PathBeneath, PathFd, PathFdError, Ruleset,
|
||||
RulesetAttr, RulesetCreatedAttr,
|
||||
RulesetAttr,
|
||||
};
|
||||
use miette::{IntoDiagnostic, Result};
|
||||
use openshell_core::policy::{LandlockCompatibility, SandboxPolicy};
|
||||
@@ -95,6 +95,12 @@ pub struct PreparedRuleset {
|
||||
compatibility: LandlockCompatibility,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
enum PathOpenMode {
|
||||
Privileged,
|
||||
CurrentUser,
|
||||
}
|
||||
|
||||
/// Phase 1: Open `PathFds` and build the Landlock ruleset **as root**.
|
||||
///
|
||||
/// This must run before `drop_privileges()` so that `PathFd::new()` can open
|
||||
@@ -103,6 +109,27 @@ pub struct PreparedRuleset {
|
||||
/// Returns `None` if there are no filesystem paths to restrict (no-op).
|
||||
/// Returns `Some(PreparedRuleset)` on success, or an error.
|
||||
pub fn prepare(policy: &SandboxPolicy, workdir: Option<&str>) -> Result<Option<PreparedRuleset>> {
|
||||
prepare_with_path_open_mode(policy, workdir, PathOpenMode::Privileged)
|
||||
}
|
||||
|
||||
/// Phase 1 for already-unprivileged workloads.
|
||||
///
|
||||
/// Kubernetes sidecar mode starts the process supervisor as the sandbox UID, so
|
||||
/// Landlock path FDs are opened as the same UID that will run the workload.
|
||||
/// Paths this UID cannot open are already unavailable to the workload; omit
|
||||
/// them from the allowlist and let the resulting ruleset deny everything else.
|
||||
pub fn prepare_current_user(
|
||||
policy: &SandboxPolicy,
|
||||
workdir: Option<&str>,
|
||||
) -> Result<Option<PreparedRuleset>> {
|
||||
prepare_with_path_open_mode(policy, workdir, PathOpenMode::CurrentUser)
|
||||
}
|
||||
|
||||
fn prepare_with_path_open_mode(
|
||||
policy: &SandboxPolicy,
|
||||
workdir: Option<&str>,
|
||||
path_open_mode: PathOpenMode,
|
||||
) -> Result<Option<PreparedRuleset>> {
|
||||
let read_only = policy.filesystem.read_only.clone();
|
||||
let mut read_write = policy.filesystem.read_write.clone();
|
||||
|
||||
@@ -188,7 +215,7 @@ pub fn prepare(policy: &SandboxPolicy, workdir: Option<&str>) -> Result<Option<P
|
||||
let mut rules_applied: usize = 0;
|
||||
|
||||
for path in &read_only {
|
||||
if let Some(path_fd) = try_open_path(path, compatibility)? {
|
||||
if let Some(path_fd) = try_open_path(path, compatibility, path_open_mode)? {
|
||||
debug!(path = %path.display(), "Landlock allow read-only");
|
||||
ruleset = ruleset
|
||||
.add_rule(PathBeneath::new(path_fd, access_read))
|
||||
@@ -198,7 +225,7 @@ pub fn prepare(policy: &SandboxPolicy, workdir: Option<&str>) -> Result<Option<P
|
||||
}
|
||||
|
||||
for path in &read_write {
|
||||
if let Some(path_fd) = try_open_path(path, compatibility)? {
|
||||
if let Some(path_fd) = try_open_path(path, compatibility, path_open_mode)? {
|
||||
debug!(path = %path.display(), "Landlock allow read-write");
|
||||
ruleset = ruleset
|
||||
.add_rule(PathBeneath::new(path_fd, access_all))
|
||||
@@ -325,7 +352,11 @@ pub fn apply(policy: &SandboxPolicy, workdir: Option<&str>) -> Result<()> {
|
||||
///
|
||||
/// In `HardRequirement` mode, any failure is fatal — the caller propagates the
|
||||
/// error, which ultimately aborts sandbox startup.
|
||||
fn try_open_path(path: &Path, compatibility: &LandlockCompatibility) -> Result<Option<PathFd>> {
|
||||
fn try_open_path(
|
||||
path: &Path,
|
||||
compatibility: &LandlockCompatibility,
|
||||
path_open_mode: PathOpenMode,
|
||||
) -> Result<Option<PathFd>> {
|
||||
match PathFd::new(path) {
|
||||
Ok(fd) => Ok(Some(fd)),
|
||||
Err(err) => {
|
||||
@@ -335,6 +366,28 @@ fn try_open_path(path: &Path, compatibility: &LandlockCompatibility) -> Result<O
|
||||
PathFdError::OpenCall { source, .. }
|
||||
if source.kind() == std::io::ErrorKind::NotFound
|
||||
);
|
||||
if matches!(path_open_mode, PathOpenMode::CurrentUser) {
|
||||
if is_not_found {
|
||||
debug!(
|
||||
path = %path.display(),
|
||||
reason,
|
||||
"Skipping non-existent Landlock path for current user"
|
||||
);
|
||||
} else {
|
||||
openshell_ocsf::ocsf_emit!(
|
||||
openshell_ocsf::ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx())
|
||||
.severity(openshell_ocsf::SeverityId::Informational)
|
||||
.status(openshell_ocsf::StatusId::Success)
|
||||
.state(openshell_ocsf::StateId::Other, "already-denied")
|
||||
.message(format!(
|
||||
"Skipping inaccessible Landlock path for current user [path:{} error:{err}]",
|
||||
path.display()
|
||||
))
|
||||
.build()
|
||||
);
|
||||
}
|
||||
return Ok(None);
|
||||
}
|
||||
match compatibility {
|
||||
LandlockCompatibility::BestEffort => {
|
||||
// NotFound is expected for stale baseline paths (e.g.
|
||||
@@ -421,6 +474,7 @@ mod tests {
|
||||
let result = try_open_path(
|
||||
&PathBuf::from("/nonexistent/openshell/test/path"),
|
||||
&LandlockCompatibility::BestEffort,
|
||||
PathOpenMode::Privileged,
|
||||
);
|
||||
assert!(result.is_ok());
|
||||
assert!(result.unwrap().is_none());
|
||||
@@ -431,6 +485,7 @@ mod tests {
|
||||
let result = try_open_path(
|
||||
&PathBuf::from("/nonexistent/openshell/test/path"),
|
||||
&LandlockCompatibility::HardRequirement,
|
||||
PathOpenMode::Privileged,
|
||||
);
|
||||
assert!(result.is_err());
|
||||
let err_msg = result.unwrap_err().to_string();
|
||||
@@ -447,11 +502,26 @@ mod tests {
|
||||
#[test]
|
||||
fn try_open_path_succeeds_for_existing_path() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let result = try_open_path(dir.path(), &LandlockCompatibility::BestEffort);
|
||||
let result = try_open_path(
|
||||
dir.path(),
|
||||
&LandlockCompatibility::BestEffort,
|
||||
PathOpenMode::Privileged,
|
||||
);
|
||||
assert!(result.is_ok());
|
||||
assert!(result.unwrap().is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn try_open_path_current_user_skips_missing_path_in_hard_requirement() {
|
||||
let result = try_open_path(
|
||||
&PathBuf::from("/nonexistent/openshell/test/path"),
|
||||
&LandlockCompatibility::HardRequirement,
|
||||
PathOpenMode::CurrentUser,
|
||||
);
|
||||
assert!(result.is_ok());
|
||||
assert!(result.unwrap().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn classify_not_found() {
|
||||
let err = std::io::Error::from_raw_os_error(libc::ENOENT);
|
||||
|
||||
@@ -12,7 +12,7 @@ use std::path::PathBuf;
|
||||
use std::sync::Once;
|
||||
|
||||
/// Opaque handle to a prepared-but-not-yet-enforced sandbox.
|
||||
/// Holds the Landlock ruleset with `PathFds` opened as root.
|
||||
/// Holds the Landlock ruleset with `PathFds` opened before child exec.
|
||||
pub struct PreparedSandbox {
|
||||
landlock: Option<landlock::PreparedRuleset>,
|
||||
policy: SandboxPolicy,
|
||||
@@ -30,6 +30,21 @@ pub fn prepare(policy: &SandboxPolicy, workdir: Option<&str>) -> Result<Prepared
|
||||
})
|
||||
}
|
||||
|
||||
/// Phase 1 for already-unprivileged workloads.
|
||||
///
|
||||
/// Opens Landlock `PathFds` as the current UID. This is used by Kubernetes
|
||||
/// sidecar mode, where the agent container already runs as the sandbox user.
|
||||
pub fn prepare_current_user(
|
||||
policy: &SandboxPolicy,
|
||||
workdir: Option<&str>,
|
||||
) -> Result<PreparedSandbox> {
|
||||
let landlock = landlock::prepare_current_user(policy, workdir)?;
|
||||
Ok(PreparedSandbox {
|
||||
landlock,
|
||||
policy: policy.clone(),
|
||||
})
|
||||
}
|
||||
|
||||
/// Phase 2: Enforce prepared sandbox restrictions (after `drop_privileges`).
|
||||
///
|
||||
/// Calls `restrict_self()` for Landlock and applies seccomp filters.
|
||||
|
||||
@@ -65,7 +65,7 @@ fn ssh_server_init(
|
||||
if let Some(parent) = listen_path.parent() {
|
||||
std::fs::create_dir_all(parent).into_diagnostic()?;
|
||||
#[cfg(unix)]
|
||||
if enforcement_mode.enforces_process_controls() && !shared_socket {
|
||||
if enforcement_mode.uses_privileged_process_setup() && !shared_socket {
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
let perms = std::fs::Permissions::from_mode(0o700);
|
||||
std::fs::set_permissions(parent, perms).into_diagnostic()?;
|
||||
@@ -827,20 +827,15 @@ fn spawn_pty_shell(
|
||||
|
||||
// Probe Landlock availability from the parent process where tracing works.
|
||||
#[cfg(target_os = "linux")]
|
||||
if enforcement_mode.enforces_process_controls() {
|
||||
if enforcement_mode.enforces_child_sandbox() {
|
||||
sandbox::linux::log_sandbox_readiness(policy, workdir.as_deref());
|
||||
}
|
||||
|
||||
// Phase 1 (as root): Prepare Landlock ruleset before drop_privileges.
|
||||
// Phase 1: Prepare Landlock ruleset before the child applies it.
|
||||
#[cfg(target_os = "linux")]
|
||||
let prepared_sandbox = if enforcement_mode.enforces_process_controls() {
|
||||
Some(
|
||||
sandbox::linux::prepare(policy, workdir.as_deref())
|
||||
.map_err(|err| anyhow::anyhow!("Failed to prepare sandbox: {err}"))?,
|
||||
)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let prepared_sandbox =
|
||||
crate::process::prepare_child_sandbox(policy, workdir.as_deref(), enforcement_mode)
|
||||
.map_err(|err| anyhow::anyhow!("Failed to prepare sandbox: {err}"))?;
|
||||
|
||||
#[cfg(unix)]
|
||||
{
|
||||
@@ -986,20 +981,15 @@ fn spawn_pipe_exec(
|
||||
|
||||
// Probe Landlock availability from the parent process where tracing works.
|
||||
#[cfg(target_os = "linux")]
|
||||
if enforcement_mode.enforces_process_controls() {
|
||||
if enforcement_mode.enforces_child_sandbox() {
|
||||
sandbox::linux::log_sandbox_readiness(policy, workdir.as_deref());
|
||||
}
|
||||
|
||||
// Phase 1 (as root): Prepare Landlock ruleset before drop_privileges.
|
||||
// Phase 1: Prepare Landlock ruleset before the child applies it.
|
||||
#[cfg(target_os = "linux")]
|
||||
let prepared_sandbox = if enforcement_mode.enforces_process_controls() {
|
||||
Some(
|
||||
sandbox::linux::prepare(policy, workdir.as_deref())
|
||||
.map_err(|err| anyhow::anyhow!("Failed to prepare sandbox: {err}"))?,
|
||||
)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let prepared_sandbox =
|
||||
crate::process::prepare_child_sandbox(policy, workdir.as_deref(), enforcement_mode)
|
||||
.map_err(|err| anyhow::anyhow!("Failed to prepare sandbox: {err}"))?;
|
||||
|
||||
#[cfg(unix)]
|
||||
{
|
||||
@@ -1157,7 +1147,7 @@ mod unsafe_pty {
|
||||
#[cfg(target_os = "linux")]
|
||||
let mut prepared = prepared;
|
||||
#[cfg(target_os = "linux")]
|
||||
let supervisor_identity_mount = if enforcement_mode.enforces_process_controls() {
|
||||
let supervisor_identity_mount = if enforcement_mode.uses_privileged_process_setup() {
|
||||
crate::process::supervisor_identity_mount_from_env().map_err(|err| {
|
||||
anyhow::anyhow!("failed to prepare supervisor identity isolation: {err}")
|
||||
})?
|
||||
@@ -1205,7 +1195,7 @@ mod unsafe_pty {
|
||||
#[cfg(target_os = "linux")]
|
||||
let mut prepared = prepared;
|
||||
#[cfg(target_os = "linux")]
|
||||
let supervisor_identity_mount = if enforcement_mode.enforces_process_controls() {
|
||||
let supervisor_identity_mount = if enforcement_mode.uses_privileged_process_setup() {
|
||||
crate::process::supervisor_identity_mount_from_env().map_err(|err| {
|
||||
anyhow::anyhow!("failed to prepare supervisor identity isolation: {err}")
|
||||
})?
|
||||
@@ -1260,7 +1250,7 @@ mod unsafe_pty {
|
||||
|
||||
// Drop privileges. initgroups/setgid/setuid need /etc/group and
|
||||
// /etc/passwd which would be blocked if Landlock were already enforced.
|
||||
if enforcement_mode.enforces_process_controls() {
|
||||
if enforcement_mode.uses_privileged_process_setup() {
|
||||
drop_privileges(policy).map_err(|err| std::io::Error::other(err.to_string()))?;
|
||||
}
|
||||
crate::process::harden_child_process()
|
||||
@@ -1275,7 +1265,7 @@ mod unsafe_pty {
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
if enforcement_mode.enforces_process_controls() {
|
||||
if enforcement_mode.enforces_child_sandbox() {
|
||||
sandbox::apply(policy, None).map_err(|err| std::io::Error::other(err.to_string()))?;
|
||||
}
|
||||
|
||||
|
||||
@@ -237,7 +237,7 @@ add `ci/values-spire.yaml` to the OpenShell release values files.
|
||||
| supervisor.image.pullPolicy | string | `""` | Supervisor image pull policy. Defaults to the gateway image pull policy when empty. |
|
||||
| supervisor.image.repository | string | `"ghcr.io/nvidia/openshell/supervisor"` | Supervisor image repository. |
|
||||
| supervisor.image.tag | string | `""` | Supervisor image tag. Defaults to the chart appVersion when empty. |
|
||||
| supervisor.proxyUid | int | `1337` | UID for the long-running network sidecar in sidecar topology. The network init container installs nftables rules that exempt this UID. |
|
||||
| supervisor.sidecar.proxyUid | int | `1337` | UID for the long-running network sidecar in sidecar topology. The network init container installs nftables rules that exempt this UID. |
|
||||
| supervisor.sideloadMethod | string | `""` | How the supervisor binary is delivered into sandbox pods. Empty (default) = auto-detect from cluster version: K8s >= v1.35 -> "image-volume" (ImageVolume enabled by default; GA in v1.36) K8s < v1.35 -> "init-container" (copies via init container + emptyDir) On K8s v1.33-v1.34 with the ImageVolume feature gate manually enabled, set this to "image-volume" explicitly. |
|
||||
| supervisor.topology | string | `"combined"` | Supervisor pod topology for Kubernetes sandboxes. "combined" runs the current single supervisor container in the agent pod. "sidecar" runs network enforcement in a dedicated sidecar and the process supervisor as a low-capability wrapper in the agent container. |
|
||||
| tolerations | list | `[]` | Tolerations for the gateway pod. |
|
||||
|
||||
@@ -113,8 +113,7 @@ data:
|
||||
grpc_endpoint = {{ include "openshell.grpcEndpoint" . | quote }}
|
||||
service_account_name = {{ include "openshell.sandboxServiceAccountName" . | quote }}
|
||||
supervisor_sideload_method = {{ include "openshell.supervisorSideloadMethod" . | quote }}
|
||||
supervisor_topology = {{ .Values.supervisor.topology | default "combined" | quote }}
|
||||
proxy_uid = {{ .Values.supervisor.proxyUid | default 1337 }}
|
||||
topology = {{ .Values.supervisor.topology | default "combined" | quote }}
|
||||
sa_token_ttl_secs = {{ .Values.server.sandboxJwt.k8sSaTokenTtlSecs | default 3600 }}
|
||||
{{- if .Values.server.providerTokenGrants.spiffe.enabled }}
|
||||
provider_spiffe_workload_api_socket_path = {{ .Values.server.providerTokenGrants.spiffe.workloadApiSocketPath | quote }}
|
||||
@@ -143,3 +142,6 @@ data:
|
||||
{{- if .Values.supervisor.image.pullPolicy }}
|
||||
supervisor_image_pull_policy = {{ .Values.supervisor.image.pullPolicy | quote }}
|
||||
{{- end }}
|
||||
|
||||
[openshell.drivers.kubernetes.sidecar]
|
||||
proxy_uid = {{ .Values.supervisor.sidecar.proxyUid | default 1337 }}
|
||||
|
||||
@@ -83,23 +83,26 @@ tests:
|
||||
path: data["gateway.toml"]
|
||||
pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?service_account_name\s*=\s*"openshell-sandbox"'
|
||||
|
||||
- it: renders supervisor topology under [openshell.drivers.kubernetes]
|
||||
- it: renders topology under [openshell.drivers.kubernetes]
|
||||
template: templates/gateway-config.yaml
|
||||
set:
|
||||
supervisor.topology: sidecar
|
||||
asserts:
|
||||
- matchRegex:
|
||||
path: data["gateway.toml"]
|
||||
pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?supervisor_topology\s*=\s*"sidecar"'
|
||||
pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?topology\s*=\s*"sidecar"'
|
||||
- notMatchRegex:
|
||||
path: data["gateway.toml"]
|
||||
pattern: 'supervisor_topology\s*='
|
||||
|
||||
- it: renders proxy uid under [openshell.drivers.kubernetes]
|
||||
- it: renders proxy uid under [openshell.drivers.kubernetes.sidecar]
|
||||
template: templates/gateway-config.yaml
|
||||
set:
|
||||
supervisor.proxyUid: 2200
|
||||
supervisor.sidecar.proxyUid: 2200
|
||||
asserts:
|
||||
- matchRegex:
|
||||
path: data["gateway.toml"]
|
||||
pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?proxy_uid\s*=\s*2200'
|
||||
pattern: '(?ms)\[openshell\.drivers\.kubernetes\.sidecar\].*?proxy_uid\s*=\s*2200'
|
||||
|
||||
- it: renders sandbox image pull secrets under [openshell.drivers.kubernetes]
|
||||
template: templates/gateway-config.yaml
|
||||
|
||||
@@ -49,9 +49,10 @@ supervisor:
|
||||
# "sidecar" runs network enforcement in a dedicated sidecar and the process
|
||||
# supervisor as a low-capability wrapper in the agent container.
|
||||
topology: "combined"
|
||||
# -- UID for the long-running network sidecar in sidecar topology. The
|
||||
# network init container installs nftables rules that exempt this UID.
|
||||
proxyUid: 1337
|
||||
sidecar:
|
||||
# -- UID for the long-running network sidecar in sidecar topology. The
|
||||
# network init container installs nftables rules that exempt this UID.
|
||||
proxyUid: 1337
|
||||
|
||||
# -- Image pull secrets attached to gateway and helper pods.
|
||||
imagePullSecrets: []
|
||||
|
||||
@@ -161,7 +161,7 @@ The most commonly changed values are:
|
||||
| `pkiInitJob.serverDnsNames` / `certManager.serverDnsNames` | Additional gateway server DNS SANs. Wildcard SANs also enable sandbox service URLs under that domain. |
|
||||
| `supervisor.sideloadMethod` | How the supervisor binary is delivered into sandbox pods. Leave empty to auto-detect based on cluster version: clusters running Kubernetes 1.35 or later use `image-volume` (ImageVolume GA in 1.36); older clusters use `init-container`. Set explicitly to `image-volume` on Kubernetes 1.33 or 1.34 with the ImageVolume feature gate enabled, or to `init-container` to force the legacy path on any version. |
|
||||
| `supervisor.topology` | Sandbox pod topology. Refer to [Topology](/kubernetes/topology). |
|
||||
| `supervisor.proxyUid` | Non-root UID for the long-running network sidecar when `supervisor.topology=sidecar`. The UID must not match the sandbox UID. |
|
||||
| `supervisor.sidecar.proxyUid` | Non-root UID for the long-running network sidecar when `supervisor.topology=sidecar`. The UID must not match the sandbox UID. |
|
||||
|
||||
Use a values file for repeatable deployments:
|
||||
|
||||
|
||||
@@ -21,7 +21,7 @@ lower-privilege agent container.
|
||||
| Topology | Use when | Main tradeoff |
|
||||
|---|---|---|
|
||||
| `combined` | You need OpenShell network, filesystem, and process controls in the sandbox workload. | The agent container carries the Linux capabilities the supervisor needs. |
|
||||
| `sidecar` | You need the agent container to run as non-root without added Linux capabilities, and network policy is the primary control. | The process supervisor is network-only, so filesystem/process controls do not run in the agent container. |
|
||||
| `sidecar` | You need the agent container to run as non-root without added Linux capabilities, and network policy is the primary control. | Privilege-dropping and supervisor mount isolation do not run in the agent container. |
|
||||
|
||||
## Privilege Model
|
||||
|
||||
@@ -121,7 +121,7 @@ The pod contains these OpenShell-managed pieces:
|
||||
| Component | Runs as | Purpose |
|
||||
|---|---|---|
|
||||
| Network init container | Root with setup capabilities | Installs pod-level nftables rules and prepares shared sidecar state. |
|
||||
| Network sidecar | `supervisor.proxyUid` | Runs the proxy, enforces network policy, owns gateway authentication and the gateway session, and writes proxy TLS plus local policy/provider snapshots. |
|
||||
| Network sidecar | `supervisor.sidecar.proxyUid` | Runs the proxy, enforces network policy, owns gateway authentication and the gateway session, and writes proxy TLS plus local policy/provider snapshots. |
|
||||
| Agent container | Resolved sandbox UID/GID | Runs the process supervisor and launches the user workload. |
|
||||
|
||||
In this topology, the agent container defaults to `runAsNonRoot: true`,
|
||||
@@ -137,10 +137,14 @@ default sidecar path.
|
||||
|
||||
<Warning>
|
||||
Sidecar mode runs the process supervisor in `network-only` mode. OpenShell still
|
||||
enforces network endpoint and L7 policy through the sidecar, but the process
|
||||
supervisor does not apply Landlock filesystem policy, process privilege
|
||||
dropping, or process/binary identity checks. Use `combined` topology when you
|
||||
need those controls in the same supervisor path.
|
||||
enforces network endpoint and L7 policy through the sidecar, and the process
|
||||
supervisor applies Landlock filesystem policy and child seccomp filters where
|
||||
the kernel/runtime supports them. The process supervisor does not perform
|
||||
root-to-sandbox privilege dropping because Kubernetes starts the container as
|
||||
the sandbox UID/GID, and it does not perform supervisor identity mount
|
||||
isolation because gateway credentials are not mounted into the agent container.
|
||||
Sidecar pods use `shareProcessNamespace: true` so the network sidecar can
|
||||
resolve workload process and binary identity through `/proc/<entrypoint-pid>`.
|
||||
</Warning>
|
||||
|
||||
## Credential Exposure
|
||||
@@ -161,9 +165,9 @@ watches for newer revisions that the network sidecar writes after settings
|
||||
polls, so future child processes can see refreshed provider env without giving
|
||||
the agent container gateway authentication material. This does not mutate the
|
||||
environment of the already-running workload entrypoint. Use `combined` topology
|
||||
when you need full process/filesystem enforcement in the same supervisor path;
|
||||
use additional runtime isolation when you need a stronger container boundary
|
||||
around network-only sidecar workloads.
|
||||
when you need the full single-supervisor enforcement path; use additional
|
||||
runtime isolation when you need a stronger container boundary around sidecar
|
||||
workloads.
|
||||
|
||||
## RuntimeClass Isolation
|
||||
|
||||
@@ -172,9 +176,10 @@ Containers when the cluster supports them. A sandboxed runtime strengthens the
|
||||
container boundary while OpenShell focuses on network policy enforcement from
|
||||
the sidecar.
|
||||
|
||||
Runtime classes do not re-enable the OpenShell filesystem and process controls
|
||||
that sidecar mode relaxes. Use them as an additional workload boundary, not as a
|
||||
replacement for the combined topology's full supervisor controls.
|
||||
Runtime classes do not re-enable the OpenShell privilege-drop or supervisor
|
||||
mount-isolation controls that sidecar mode relaxes. Use them as an additional
|
||||
workload boundary, not as a replacement for the combined topology's full
|
||||
supervisor controls.
|
||||
|
||||
You can set a default runtime class in the Kubernetes driver configuration or
|
||||
override it per sandbox with driver config:
|
||||
@@ -191,7 +196,9 @@ For direct gateway TOML configuration, set the Kubernetes driver fields:
|
||||
|
||||
```toml
|
||||
[openshell.drivers.kubernetes]
|
||||
supervisor_topology = "sidecar"
|
||||
topology = "sidecar"
|
||||
|
||||
[openshell.drivers.kubernetes.sidecar]
|
||||
proxy_uid = 1337
|
||||
```
|
||||
|
||||
@@ -204,11 +211,12 @@ When the Helm chart renders `gateway.toml`, set the equivalent chart values:
|
||||
```yaml
|
||||
supervisor:
|
||||
topology: sidecar
|
||||
proxyUid: 1337
|
||||
sidecar:
|
||||
proxyUid: 1337
|
||||
```
|
||||
|
||||
Leave `supervisor_topology` unset, or set it to `combined`, to keep the
|
||||
original single-container supervisor path. For Helm installs, leave
|
||||
Leave `topology` unset, or set it to `combined`, to keep the original
|
||||
single-container supervisor path. For Helm installs, leave
|
||||
`supervisor.topology` unset or set it to `combined`.
|
||||
|
||||
## Next Steps
|
||||
|
||||
@@ -180,10 +180,7 @@ supervisor_sideload_method = "image-volume"
|
||||
# "combined" runs the existing single supervisor container with full process,
|
||||
# filesystem, and network enforcement in the agent container. "sidecar" moves
|
||||
# pod-level network enforcement and gateway session handling into a network sidecar.
|
||||
supervisor_topology = "combined"
|
||||
# UID used by the long-running network sidecar. In sidecar topology the
|
||||
# network init container installs nftables rules that exempt this UID.
|
||||
proxy_uid = 1337
|
||||
topology = "combined"
|
||||
grpc_endpoint = "https://openshell-gateway.agents.svc:8080"
|
||||
ssh_socket_path = "/run/openshell/ssh.sock"
|
||||
client_tls_secret_name = "openshell-client-tls"
|
||||
@@ -207,6 +204,11 @@ provider_spiffe_workload_api_socket_path = "/spiffe-workload-api/spire-agent.soc
|
||||
# back to 1000 on non-OpenShift clusters.
|
||||
# sandbox_uid = 1500
|
||||
# sandbox_gid = 1500
|
||||
|
||||
[openshell.drivers.kubernetes.sidecar]
|
||||
# UID used by the long-running network sidecar. In sidecar topology the
|
||||
# network init container installs nftables rules that exempt this UID.
|
||||
proxy_uid = 1337
|
||||
```
|
||||
|
||||
### Docker
|
||||
|
||||
@@ -306,8 +306,8 @@ For maintainer-level implementation details, refer to the [Kubernetes driver REA
|
||||
| `supervisor_image` | `supervisor.image.repository` / `supervisor.image.tag` | Set the supervisor image that provides the `openshell-sandbox` binary. |
|
||||
| `supervisor_image_pull_policy` | `supervisor.image.pullPolicy` | Set the Kubernetes image pull policy for the supervisor image. |
|
||||
| `supervisor_sideload_method` | `supervisor.sideloadMethod` | How the supervisor binary is delivered into sandbox pods. Leave empty to auto-detect from cluster version. Set to `image-volume` to mount the supervisor OCI image directly as a volume (requires Kubernetes 1.33+ with the ImageVolume feature gate; GA in 1.36), or `init-container` to copy it through an init container on older clusters. |
|
||||
| `supervisor_topology` | `supervisor.topology` | Set `combined` for the default single supervisor path, or `sidecar` to move pod-level network enforcement and the gateway session into a dedicated sidecar. |
|
||||
| `proxy_uid` | `supervisor.proxyUid` | UID used by the long-running network sidecar in `sidecar` topology. The network init container exempts this UID from proxy redirection. |
|
||||
| `topology` | `supervisor.topology` | Set `combined` for the default single supervisor path, or `sidecar` to move pod-level network enforcement and the gateway session into a dedicated sidecar. |
|
||||
| `sidecar.proxy_uid` | `supervisor.sidecar.proxyUid` | UID used by the long-running network sidecar in `sidecar` topology. The network init container exempts this UID from proxy redirection. |
|
||||
| `app_armor_profile` | `server.appArmorProfile` | Set the sandbox agent container's AppArmor profile. Helm defaults this to `Unconfined` so AppArmor-enabled nodes do not block supervisor network namespace setup. Set the Helm value to an empty string to omit the field, or use `RuntimeDefault` or `Localhost/<profile-name>` for operator-managed profiles. |
|
||||
| `workspace_default_storage_size` | `server.workspaceDefaultStorageSize` | Set the default workspace PVC size for new sandboxes. |
|
||||
| `sa_token_ttl_secs` | `server.sandboxJwt.k8sSaTokenTtlSecs` | Set the projected ServiceAccount token TTL used for the bootstrap token exchange. |
|
||||
@@ -324,10 +324,12 @@ sandbox bootstrap token or client TLS secret in the default sidecar path. The
|
||||
provider environment snapshot is refreshed by the network sidecar after
|
||||
settings polls so future child processes can see updated provider env without
|
||||
gateway access in the agent container.
|
||||
Sidecar mode keeps gateway session and SSH behavior, but the process supervisor
|
||||
runs in `network-only` mode: filesystem policy, process privilege dropping, and
|
||||
process/binary identity checks are not applied by the process container. Network
|
||||
policy still runs in the sidecar and remains endpoint/L7 scoped.
|
||||
Sidecar mode keeps gateway session and SSH behavior. The process supervisor
|
||||
applies Landlock filesystem policy and child seccomp filters where supported,
|
||||
but it does not perform root-to-sandbox privilege dropping or supervisor
|
||||
identity mount isolation. Network policy still runs in the sidecar, and sidecar
|
||||
pods set `shareProcessNamespace: true` so the network sidecar can resolve
|
||||
process/binary identity through `/proc/<entrypoint-pid>`.
|
||||
|
||||
The Kubernetes driver creates namespaced `agents.x-k8s.io` `Sandbox` resources from the Kubernetes SIG Apps [agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) project. It detects the served Sandbox API at runtime, caches the selected API version for the gateway process, and uses `v1beta1` when available before falling back to `v1alpha1`, so supported Agent Sandbox installations work without version-specific operator configuration. The Agent Sandbox controller turns those resources into sandbox pods and related storage.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user