From f7ec2cfee83a2b680d57ff4a7aa3073bc3cf0386 Mon Sep 17 00:00:00 2001 From: Jordan Ganoff Date: Fri, 3 Jul 2026 09:35:06 -0700 Subject: [PATCH] revised based on feedback: - Introduced topology nomenclature to align with how we've been talking about this in other conversations - Simplified the explanation in the proposal section - Clarified binary identity - Simplified the initial contract to require the minimal interface necessary to satisfy all known topologies - Confirmed this will work with the proposed mxc (RFC 0013) proposal --- rfc/0012-isolation-backend/README.md | 468 ++++++++---------- .../codebase-grounding.md | 4 +- rfc/0012-isolation-backend/topology-matrix.md | 128 ++--- 3 files changed, 236 insertions(+), 364 deletions(-) diff --git a/rfc/0012-isolation-backend/README.md b/rfc/0012-isolation-backend/README.md index 448d5d14c..a863adbae 100644 --- a/rfc/0012-isolation-backend/README.md +++ b/rfc/0012-isolation-backend/README.md @@ -16,179 +16,192 @@ links: ## Summary -OpenShell confines untrusted agent code with network, filesystem, syscall, and identity controls. Today the supervisor builds that boundary from the agent's container, placing isolation privilege beside the code it confines. This RFC makes the boundary a pluggable **Isolation Backend**. +OpenShell confines untrusted agent code with network, filesystem, syscall, and binary-identity controls. Today the supervisor process builds that boundary from inside the agent's container, placing isolation privilege beside the code it confines. This RFC moves boundary construction and operation behind one pluggable contract: the **Isolation Backend**. -The driver still provisions the sandbox. The supervisor remains the policy authority, audit source, and agent's only egress. The backend establishes and operates the boundary and supplies connection identity through one supervisor-facing contract. +The compute driver provisions the topology. The supervisor operates each active boundary through the Isolation Backend contract. Network mediation receives all workload egress, applies network policy, and records decisions. The topology may place these functions together or in separate trusted components. -That contract supports today's in-pod boundary and future sidecar, node, split-pod, and VM implementations. Each delegated placement is separate design and implementation work, not a supervisor fork. +The contract supports today's in-pod implementation path and delegated topologies behind the same interface. Each delegated backend is separate design and implementation work, not a supervisor fork. ## Motivation -Boundary construction is embedded in the supervisor, so moving it anywhere else means changing the supervisor. One inline choice creates three problems: +Boundary construction is embedded in the supervisor, so moving it anywhere else means changing the supervisor. That placement creates three problems: - A compromise reaches the boundary-building privilege in the same container. -- Restricted and multi-tenant clusters reject the deployment. Building the boundary inline needs added Linux capabilities in the agent container (seven today, including `NET_ADMIN` and `SYS_ADMIN`); the `restricted` Pod Security Standards profile those clusters require permits no added capability beyond `NET_BIND_SERVICE`, and five of the seven (`NET_ADMIN`, `SYS_ADMIN`, `SYS_PTRACE`, `SYSLOG`, `DAC_READ_SEARCH`) are off the `baseline` allowlist too, so the deployment fails `baseline` as well. (Tracked in #899.) +- Restricted and multi-tenant clusters reject the deployment: building the boundary inline needs added Linux capabilities in the agent container that the `restricted` and `baseline` Pod Security Standards profiles refuse. The current capability set is pinned in [codebase-grounding.md](./codebase-grounding.md). (Tracked in #899.) - Each new placement adds another branch to the supervisor. All three come from coupling boundary construction to boundary operation. A common interface lets deployments move privilege without changing the supervisor. ## Non-goals -- **Implementing a delegated backend.** Each placement requires its own design and implementation. -- **Defining remote confirmation.** This contract requires confirmation; each cross-kernel backend defines its attestation mechanism. +- **Implementing a delegated backend.** Each topology requires its own design and implementation. - **Defining global authorization.** A separate proposal owns supervisor authorization. A delegated backend must still authenticate callers and scope them to one boundary. -- **Sharing supervisors.** This RFC keeps one supervisor per agent. +- **Standardizing backend-internal component coordination.** A backend may coordinate helper, sidecar, or network-mediation processes behind one lifecycle; how those components cooperate is backend-specific, not contract surface. ## Proposal -This RFC standardizes backend selection, lifecycle ordering, runtime interfaces, failure behavior, and invariants. How a backend constructs its boundary or transports these operations is implementation-specific. +Trusted deployment configuration and admission select a topology. The gateway asks a compute driver to create a sandbox environment; the driver provisions the selected topology and supplies the `TopologyDescriptor` to the supervisor. The supervisor resolves the descriptor's backend and calls it instead of platform-specific kernel primitives. The backend binds the trusted sandbox context, exposes network-mediation and process operations, confirms standing enforcement (the controls established independently of any workload process), and ensures the per-process launch-time controls are in force before agent code starts. -The gateway and driver provision the sandbox on the control plane. The supervisor drives the backend while the agent runs. +In the in-pod topology, the supervisor drives a backend implemented in the same process. Other topologies may delegate backend operations to helpers or shared services without changing the supervisor lifecycle. + +This RFC uses five terms: + +- **Topology:** the placement and arrangement provisioned by the compute driver. +- **Isolation boundary:** the effective network, filesystem, syscall, and binary-identity enforcement around the workload. +- **Supervisor:** the trusted role that operates an active boundary through the Isolation Backend contract, from `attach` until termination. Each active boundary has exactly one logical supervisor. The in-sandbox supervisor process hosts this role today; another topology may host it in another trusted component. +- **Isolation Backend:** the implementation selected by the topology descriptor and driven by the supervisor. +- **Network mediation:** the trusted function that receives outbound connections from processes inside the boundary, evaluates network policy using backend-supplied binary identity, forwards allowed traffic, and records network policy decisions. It may run with the supervisor or in another trusted component. ```mermaid -graph TB - subgraph ControlPlane["Control plane (at sandbox creation)"] - direction TB - Gateway["Gateway"] - Driver["Driver"] - Gateway -->|provisions| Driver - end +flowchart TB + Gateway["Gateway"] -->|"create sandbox"| Driver["Compute driver"] - subgraph DataPlane["Data plane (as the agent runs)"] - direction TB - subgraph Boundary["The boundary"] - direction TB - Agent["Agent"] + subgraph Topology["Driver-provisioned topology (placement varies)"] + Supervisor["Supervisor"] + Backend["Isolation Backend (may coordinate components)"] + Mediator["Network mediation"] + subgraph Boundary["Isolation boundary"] + Workload["Workload"] end - Supervisor["Supervisor (the mediator)"] - Runtime["Isolation Backend (runtime contract; location varies)"] - Egress["Egress"] - Agent ==>|only egress: the proxy| Supervisor - Supervisor -->|allowed egress| Egress - Supervisor -->|drives runtime contract| Runtime + + Supervisor -->|"drives contract"| Backend + Backend -->|"establishes and confirms"| Boundary + Backend -->|"after Ready: makes agent runnable under launch controls"| Workload + Workload ==>|"only egress"| Mediator end - Driver ==>|provisions resources + descriptor| Boundary - Runtime -->|establishes, confirms, operates| Boundary - - style Boundary fill:none,stroke:#9575cd,color:#00084D - style Gateway fill:#b39ddb,stroke:#9575cd,color:#fff - style Supervisor fill:#b39ddb,stroke:#9575cd,color:#fff - style Driver fill:#a5d6a7,stroke:#81c784,color:#00084D - style Runtime fill:#a5d6a7,stroke:#81c784,color:#00084D - style Agent fill:#a5d6a7,stroke:#81c784,color:#384D54 - style Egress fill:#a5d6a7,stroke:#81c784,color:#00084D + Driver -->|"resources + TopologyDescriptor"| Supervisor + Mediator -->|"allowed egress"| Egress["Egress"] ``` -The agent is contained behind the boundary; the supervisor's proxy is its only egress. +A boundary is active from successful `attach` until its workload terminates and the backend releases the binding. A backend may coordinate multiple trusted helper or network-mediation processes for that boundary. `Ready` means every required component has reached its pre-launch condition. What the driver does with the resource afterward is outside this contract. -### Provisioning the boundary +### Contract invariants -Provisioning runs at sandbox creation, on the control plane, in a fixed order: +Six invariants hold for every boundary: -1. **Admission resolves the backend** from deployment config, not `SandboxPolicy`. Resolution fails closed; there is no downgrade. -2. **The driver provisions the compute resource** and anything the selected backend needs before the supervisor starts. -3. **The backend establishes the boundary before untrusted code runs.** This may happen during provisioning or `attach`, depending on the backend. +1. Workload egress is denied except through network mediation for the boundary's lifetime. +2. No untrusted instruction executes until every admitted control applicable to that process is in force. +3. The backend and network mediation authorize an operation only when the complete effective policy permits it; there is no silent weakening. +4. Agent activation, `exec`, and forwarding occur only through the active backend, and every workload process remains in the compute driver's provisioned execution environment. +5. Shared infrastructure preserves strict per-boundary lifecycle, policy, identity, enforcement, and cleanup isolation. +6. Loss of required enforcement ends `Running` and terminates all workload processes. Network-mediation unavailability denies outbound connections and never enables direct egress. -The driver provisions resources and sets the descriptor; the supervisor reads it and drives the contract. Each backend design owns its construction, authorization, tenant isolation, and orphan reclamation. +### Provisioning -### The boundary descriptor +Provisioning runs on the control plane, and three rules hold in every topology: -The driver sets `OPENSHELL_BOUNDARY_DESCRIPTOR` for every sandbox, including in-pod. The common, size-bounded envelope names the backend and carries an opaque attachment payload. Common code retrieves trusted requirements from the authenticated admission record, verifies the descriptor against them, and constructs `VerifiedBoundaryDescriptor`; the factory then validates payload semantics and any endpoint authentication, freshness, or replay requirements. Missing or invalid input fails closed with no default backend. The supervisor does not pass the descriptor to the workload. All drivers and the supervisor use the same versioned boundary descriptor representation. +1. **Admission selects the topology** from trusted deployment configuration, not `SandboxPolicy`. The resulting `TopologyDescriptor` names the required backend, and resolution never falls back to another backend. +2. **The compute driver provisions the topology** and anything the selected backend needs. +3. **The backend establishes standing enforcement before untrusted code runs**, during provisioning or `attach`, depending on the backend. + +A compute driver may provision a resource and `TopologyDescriptor` before the control plane assigns it to a sandbox. No untrusted workload runs while the resource is unassigned. After claim or assignment produces a trusted `SandboxContext`, the supervisor calls `attach`; the backend either binds that context to the prepared resource and returns `Bound`, or rejects it as incompatible. Pool creation, claim, reset, release, and recycling remain outside this contract. + +### The topology descriptor + +The driver supplies a descriptor for every provisioned topology, including in-pod and resources prepared before assignment. The common envelope names the backend and carries an opaque payload. ```rust -struct BoundaryDescriptor { - version: u32, - backend_id: String, - payload: Vec, // backend-specific attachment data -} - -// Trusted expectations from the authenticated admission record, retained by -// VerifiedBoundaryDescriptor after common verification. -struct AdmittedBoundaryRequirements { - backend_id: String, +struct TopologyDescriptor { contract_version: u32, - policy_digest: Hash, - required_confirmation: ConfirmationKind, - minimum_identity: Assurance, + backend_name: String, + payload: Vec, } ``` -The envelope version and backend contract version are distinct. Only common verification can construct `VerifiedBoundaryDescriptor`, which has no public constructor. An exact contract version implies support for every field of that `SandboxPolicy` version; confirmation kind and identity assurance are the only negotiated security capabilities. +`contract_version` is the Isolation Backend contract version. The descriptor is transport-neutral. Provisioning supplies it to the supervisor before `attach`; how it is transported is topology-specific and outside this contract, and every transport preserves one property: workload-controlled input cannot select or modify the descriptor. -Every backend confirms effective enforcement before returning `Ready`. A local backend may observe it directly; a remote backend may return verified attestation. Remote evidence is fresh, bound to the verified descriptor, authenticated claim, and admitted policy digest, and verified against a trust root outside the workload's adversary domain; its protocol remains backend-specific. Mutable enforcement remains confirmed for the boundary's lifetime, and drift fails the sandbox. +The opaque payload identifies, or gives the backend enough information to resolve, the exact driver-provisioned resource. It may also carry topology-specific endpoint or helper-role information; there are no common topology or role fields. -Attachment and sandbox identity binding are separate. A warm-pooled boundary is established against its admitted image baseline before a sandbox exists; `claim` later binds the sandbox, policy, agent, and resources. An incompatible claim takes the cold path. Before claim, only digest-pinned OpenShell-controlled placeholder code may run. +Common verification requires: -### The runtime contract +- the descriptor's `backend_name` matches the backend required by the admitted topology; +- the descriptor and resolved factory versions equal the supervisor-supported contract version; and +- `SandboxContext` is constructed after the control plane assigns the resource to the admitted sandbox, using authenticated control-plane and trusted supervisor state. -The boundary spans four dimensions: network, filesystem, syscall, and identity. The in-pod backend realizes them as a network namespace and routing, Landlock, seccomp, and procfs; other backends realize them differently. The contract adds no enforcement of its own; it standardizes how the supervisor drives whichever backend a deployment admits. +The supervisor validates the descriptor's common fields and resolves its `backend_name` without fallback. `VerifiedTopologyDescriptor` does not imply that the opaque payload is valid; the selected backend validates it and atomically binds the provisioned resource to the trusted `SandboxContext` during `attach`. Any failure rejects the sandbox. -A backend registers a factory under a `backend_id`. The supervisor resolves the factory, attaches to the admitted boundary, and drives it through a fixed sequence of states. Each transition consumes the prior state, so the supervisor cannot skip a stage or invoke a later transition through an earlier handle. The Rust names below are illustrative; the states and their semantics are normative. +### The lifecycle + +The contract adds no enforcement of its own; it standardizes how the supervisor drives whichever backend a deployment admits. + +A backend registers a factory under a `backend_name`. The supervisor attaches to the admitted topology and drives the boundary through a fixed sequence of states. Each transition consumes the prior state, so the supervisor cannot skip a stage or invoke a later transition through an earlier handle. The Rust names are illustrative; the states and their semantics are normative. + +```text +attach topology + sandbox context -> Bound -> confirm -> Ready -> start_agent -> Running +``` ```rust #[async_trait] trait IsolationBackendFactory: Send + Sync { - fn backend_id(&self) -> &str; - fn capabilities(&self) -> BackendCapabilities; - async fn attach(&self, descriptor: VerifiedBoundaryDescriptor) - -> Result, BackendError>; + fn backend_name(&self) -> &str; + fn contract_version(&self) -> u32; + + async fn attach( + &self, + descriptor: VerifiedTopologyDescriptor, + sandbox: SandboxContext, + ) -> Result, BackendError>; +} + +struct SandboxContext { + sandbox_id: SandboxId, + policy: SandboxPolicy, + agent: AgentSpec, } #[async_trait] -trait BoundaryControl: Send + Sync { // retained by the supervisor after attach - async fn wait_terminated(&self) -> Result<(), BackendError>; - async fn shutdown(&self) -> Result<(), BackendError>; // idempotent +trait BoundBoundary: Send { + fn network_mediation_ingress(&self) -> Arc; + + async fn confirm( + self: Box, + ) -> Result, BackendError>; } #[async_trait] -trait AttachedBoundary: Send { // the admitted boundary; no claimed or untrusted workload - fn control(&self) -> Arc; - async fn claim(self: Box, claim: ClaimContext) -> Result, BackendError>; +trait ReadyBoundary: Send { + async fn start_agent( + self: Box, + ) -> Result, BackendError>; } + #[async_trait] -trait ClaimedBoundary: Send { // bound to sandbox identity, policy, agent, resources - async fn bind(self: Box) -> Result, BackendError>; -} -#[async_trait] -trait BoundBoundary: Send { // mediation interfaces connected - fn mediation_ingress(&self) -> Arc; - fn identity_source(&self) -> Arc; - fn events(&self) -> Arc; - async fn confirm(self: Box) -> Result, BackendError>; -} -#[async_trait] -trait ReadyBoundary: Send { // enforcement and mediation confirmed - async fn start_agent(self: Box) -> Result, BackendError>; -} -#[async_trait] -trait RunningBoundary: Send + Sync { // the agent is running behind the boundary +trait RunningBoundary: Send + Sync { fn agent(&self) -> Arc; fn exec(&self) -> Arc; fn port_forward(&self) -> Arc; } +``` +`AgentSpec` carries the complete admitted agent launch specification, including command, arguments, working directory, timeout, and interactive mode. + +`SandboxContext` carries the create-time policy baseline. After `Running`, network mediation continues to receive authorized policy revisions through its update path and applies them atomically to subsequent decisions. A revision that would change standing enforcement or launch-time controls is rejected. + +The states have normative meanings: + +- **Bound:** the topology descriptor and trusted sandbox context are bound to the same resource, and the network-mediation ingress is available. No untrusted workload code is running. +- **Ready:** the supervisor has initialized network mediation, the backend has confirmed standing enforcement, and the backend is prepared to ensure the admitted launch-time controls are in force before untrusted execution. +- **Running:** `start_agent` has made the admitted agent runnable and returned `RunningBoundary`. Every applicable launch-time control was in force before the first untrusted instruction. Whether the backend creates the agent process or releases a held, driver-provisioned execution object is backend-specific; the contract fixes the ordering, not the mechanism. + +`attach` rejects a resource already bound to an active boundary. A backend that cannot enforce the complete admitted policy fails `attach` or `confirm`. How a backend confirms its `Ready` conditions is implementation-specific. + +**Standing enforcement** is the set of controls established independently of a workload process. **Launch-time controls** are the per-process controls that must be in force before that process executes its first untrusted instruction. Both `start_agent` and every `BoundaryExec::exec` ensure this ordering for the process they start, and both preserve the provisioned execution environment. Which component applies a launch-time control is backend-specific; only the ordering is normative. If either operation does not return its success handle — including an unknown outcome — no untrusted process from that attempt may remain running. + +`start_agent` is the sole operation that may make the admitted agent runnable. A driver-provisioned execution object is conformant only if it is bound to the active `SandboxContext` and admitted `AgentSpec`, cannot execute workload-controlled code before release, and carries no pre-launch authority that can bypass admitted controls. A cooperative wait inside workload-controlled agent code is not conformant. + +`RunningBoundary::agent()` returns a handle for the admitted agent process. Processes started through `BoundaryExec` run in the same boundary and have their own process handles. Every workload process remains within the provisioned execution environment. Any exit of the admitted agent ends `Running`; the backend then terminates every remaining workload process within that environment and rejects further runtime operations, except `wait`, which returns each process's stable result. + +### Runtime operations + +```rust #[async_trait] trait BoundaryProcess: Send + Sync { // the agent, or a process started via exec async fn wait(&self) -> Result; // one stable result across repeated calls async fn signal(&self, signal: BoundarySignal) -> Result<(), BackendError>; - async fn terminate(&self) -> Result<(), BackendError>; // owns the whole descendant tree + async fn terminate(&self) -> Result<(), BackendError>; // this process and its descendants } -``` -| State | Meaning | Available to the supervisor | -|---|---|---| -| **Attached** | the admitted boundary is attached; no claimed or untrusted workload | boundary control | -| **Claimed** | bound to sandbox identity, policy, agent spec, and resources | boundary control | -| **Bound** | mediation interfaces are connected | control, mediation ingress, identity, events | -| **Ready** | enforcement and mediation are confirmed | control, agent start | -| **Running** | the agent is started behind the boundary | control, retained mediation handles, agent, `exec`, `connect` | - -The state order is structural: `attach` identifies the admitted boundary, `claim` introduces sandbox identity and workload requirements, only `ReadyBoundary` may start the agent, and only `RunningBoundary` exposes `exec` and `connect`. A pooled boundary may remain `Attached` until claimed. The supervisor retains `BoundaryControl` and owned runtime handles across consuming transitions. - -The runtime interfaces are small and owned: - -```rust #[async_trait] trait BoundaryExec: Send + Sync { async fn exec(&self, spec: ExecSpec) -> Result; @@ -207,223 +220,152 @@ trait BoundaryPortForward: Send + Sync { async fn connect(&self, target: LoopbackTarget) -> Result; } -trait EventSource: Send + Sync { - fn subscribe(&self) -> Result; -} - -#[async_trait] -trait MediationIngress: Send + Sync { // how the proxy receives connections - async fn accept(&self) -> Result; -} -struct MediatedConnection { - stream: BoundaryDuplexStream, // the workload connection - flow: Flow, // its identity token (see Identity) -} - #[async_trait] trait BoundaryTerminal: Send + Sync { async fn resize(&self, cols: u16, rows: u16) -> Result<(), BackendError>; } ``` -`MediationIngress` is the proxy's backend-neutral source of workload connections. In-pod wraps the proxy's local listener; delegated backends use their private transport. `ExecSpec` carries command, arguments, environment, working directory, and PTY settings. Streams are owned, non-PTY stdout and stderr remain separate, and PTYs support resize. Port forwarding accepts only validated loopback targets. Event loss, exit status, and signals are explicit and placement-neutral; a local PID is never the process handle. `BoundaryControl::wait_terminated` reports boundary loss separately from agent exit. +`ExecSpec` carries command, arguments, environment, working directory, and PTY settings. Streams are owned, non-PTY stdout and stderr remain separate, and PTYs support resize. Port forwarding accepts only validated loopback targets. Exit status and signals are explicit and placement-neutral; a local PID is never the process handle. These operations carry the existing agent, SSH, exec, and forwarding paths behind the contract, and all of them are mandatory conformance. -The claim carries only the common content every backend needs: +### Network mediation ```rust -struct ClaimContext { - sandbox_id: SandboxId, - policy: SandboxPolicy, // all four dimensions - agent: AgentSpec, - resource_binding: ResourceBinding, +#[async_trait] +trait NetworkMediationIngress: Send + Sync { + async fn accept(&self) -> Result; +} + +struct MediatedConnection { + stream: BoundaryDuplexStream, + binary_identity: Result, } ``` -`claim` verifies these fields against the descriptor, admission record, and policy digest. A claim is authenticated, fresh, single-use, and bound to the boundary and sandbox; the backend design defines that mechanism. `ResourceBinding` is opaque, versioned driver data for the admitted cgroup, runtime security context, and devices. The agent and every `exec` descendant remain within it. +`NetworkMediationIngress` is the per-boundary source of outbound connections consumed by network mediation. Its transport and placement are backend-specific. The backend authoritatively associates each returned connection with the active boundary without relying solely on workload-provided data. -### Supervisor integration +A `MediatedConnection` is an outbound connection from any process inside the boundary, delivered to network mediation before policy evaluation. Each carries the connection's byte stream and binary-identity resolution result. A failed `accept` yields no connection; the caller may retry. -The supervisor keeps an in-tree registry from `backend_id` to factory. Adding a backend adds an implementation and registration, not branches in the lifecycle, proxy, SSH, or session code. Delegated transports remain private to their factories. +Shared implementations isolate each boundary's state and enforcement. Failure or teardown of one boundary cannot weaken another. Network-mediation unavailability never enables direct egress. + +### Binary identity + +The backend resolves executable identity for every accepted connection and delivers the result on `MediatedConnection`, before network mediation evaluates policy. + +```rust +struct BinaryIdentity { + binary_path: PathBuf, // absolute executable path + binary_sha256: Option, + ancestors: Vec, // nearest first + cmdline_paths: Vec, // diagnostic context; never authorizes +} +``` + +If binary identity cannot be resolved, the connection is denied. A missing digest is represented as `None`. How the backend resolves identity is implementation-specific. + +Binary identity is mandatory conformance: RFC 0002 makes it part of the outbound-policy baseline. There is no capability flag and no mode that exempts a backend from resolving identity. + +### The supervisor sequence + +The supervisor keeps an in-tree registry from `backend_name` to factory. Adding a backend adds an implementation and a registration, not branches in the lifecycle, proxy, SSH, or session code. Delegated transports remain private to their factories. The supervisor runs the same sequence for every backend: -1. Read and validate the descriptor envelope. -2. Match its `backend_id` to admission. -3. Resolve the registered factory and check its capabilities. -4. Attach to the boundary and retain its `BoundaryControl`. -5. Claim the sandbox identity, policy, agent spec, and resources. -6. Bind mediation ingress and identity into the proxy, and events into the orchestrator. -7. Start mediation services. -8. Confirm readiness. -9. Start the agent. -10. Enable `exec` and port forwarding. -11. Wait for the agent or `BoundaryControl` to terminate. -12. Call `BoundaryControl::shutdown`. -13. Surface the result for platform-specific cleanup. - -A backend reports what it can do, so admission can reject a workload it cannot place: - -```rust -enum ConfirmationKind { Direct, Attested } - -struct BackendCapabilities { - contract_version: u32, - placement: BackendPlacement, - confirmation: Vec, - maximum_identity: Assurance, -} -``` - -The backend ID and exact contract version must agree across admission, descriptor, and factory. The backend must support the required confirmation kind and minimum identity assurance. Any mismatch fails closed. All other contract behavior is mandatory conformance, not a negotiable capability. +1. Obtain the `TopologyDescriptor` and trusted `SandboxContext`. +2. Verify the descriptor and resolve its `backend_name` without fallback. +3. Call `attach` to obtain `Bound`. +4. Initialize network mediation with the boundary's `NetworkMediationIngress`. +5. Call `confirm` to obtain `Ready`, then `start_agent` to obtain `Running`. +6. Use the returned runtime handles for agent wait, `exec`, and port forwarding while network mediation consumes outbound connections. ### Failure semantics -A failure is any case where the supervisor cannot obtain or apply a valid result. The contract fixes the behavior; backend-specific retry limits and transport recovery are implementation details. Every failure carries a machine-readable kind: +Every failure carries a machine-readable kind for supervisor status mapping: ```rust enum BackendErrorKind { Invalid, Denied, Unavailable, Failed, Terminated } ``` -`Invalid` covers descriptor, version, backend, and capability mismatches; `Denied` covers authenticated attachment or claim rejection. The failing operation and backend diagnostics remain structured context. +`Invalid` covers descriptor, version, and backend mismatches; `Denied` covers authenticated attachment rejection; `Unavailable` covers transient inability to serve an operation; `Failed` covers other backend faults; and `Terminated` reports abnormal boundary or workload loss, or an operation against an inactive boundary. An error never advances the lifecycle or authorizes an operation, and backend selection never falls back. -| Failure | Required behavior | -|---|---| -| Missing, malformed, unsupported, or mismatched descriptor | Fail closed; never select another backend | -| Attachment denied or verification failed | Terminal for this sandbox instance | -| Boundary unavailable during attachment | Retry the same backend only; never downgrade | -| Confirmation failed | Do not start workload code; surface the failure for cleanup | -| Claim, bind, start, or runtime operation failed | Terminal; fail closed and clean up | -| Boundary terminates after start | Stop new `exec` and `connect`, and fail the sandbox | +A backend may retry backend-private work within one `attach` call. The supervisor calls `attach` at most once per provisioned topology. If it does not return `Bound`—including an `Unavailable` or unknown outcome—the supervisor deprovisions that topology rather than reusing it or returning it to a pool. A later attempt starts with a newly provisioned descriptor. -Only `Unavailable` from `attach` is automatically retryable, always against the same backend and boundary. Mutating transitions reconcile a lost response or fail closed; they never create a second claim or agent. Failed `attach` cleans itself. After attachment, the retained `BoundaryControl` owns idempotent cleanup on every terminal path, while driver-provisioned resources remain driver-owned; backend orphan reclamation covers supervisor death. Boundary failure disables new `exec` and `connect`, drains or securely quarantines the workload, and fails the sandbox. Agent and `exec` termination cover their complete descendant trees. +Failure observation needs no separate watcher: -### Backend invariants +- an `attach` or `confirm` failure, or network-mediation initialization failure while `Bound`, prevents untrusted workload execution and causes the driver to reclaim the topology; +- if `start_agent` does not return `Running`, no untrusted process from that attempt remains, and the driver reclaims the topology; +- after `Running`, backend-detected loss of required enforcement ends `Running`, terminates all workload processes, and makes the agent process's `wait` operation return `Terminated`; +- network-mediation errors yield no authorized connection and do not by themselves end `Running`; and +- retained runtime handles reject new operations after `Running` ends, except `wait`, which returns its stable result. -Every backend satisfies these invariants: +On normal agent exit, `wait` returns the stable exit status; after `Running` ends, the compute driver may deprovision the topology. -1. **No unguarded workload egress.** Before untrusted execution and until the execution domain is drained, the workload reaches no egress but the proxy. Enforcement is effective default-deny across every protocol the platform supports, including IPv4, IPv6, raw and packet sockets, and ingress. The backend confirms behavior, not merely installed rules, and does not assume a CNI or host route provides the deny. -2. **No untrusted workload execution before `Ready`.** Workload code includes the agent image, workspace, mounts, and image-provided init code. Only digest-pinned OpenShell content may run before claim and `Ready`. Typestate enforces ordering before `Running`; calls made after dynamic termination fail closed. -3. **No unattributed workload execution.** Agent start, `exec`, and `connect` require a bound claim. A warm pool runs only its trusted placeholder before claim. Supervisor adoption removes direct launch paths that bypass the contract. -4. **Preserve the driver's execution domain.** Every workload descendant remains within the admitted cgroup, runtime security context, and device allocation carried by `ResourceBinding`. A privileged helper sees workload devices only through an admission-visible, audited authorization. -5. **No silent weakening.** A backend implements every admitted `SandboxPolicy` field across network, filesystem, syscall, and identity, or rejects the workload at admission or claim. +Component restart handling is backend-specific. An active boundary must retain one logical supervisor and continue to satisfy the contract's binding and enforcement guarantees. -The invariants hold until the execution domain is drained. Landlock and seccomp are monotonic; mutable routing and firewall state stays fenced from the workload and preserves default-deny atomically. Loss of enforcement, identity, events, or backend control rejects new operations, drains or quarantines the workload, and fails the sandbox. Shared-kernel backends do not claim to contain a kernel compromise. +### Topologies -### Identity - -The proxy calls `IdentitySource::resolve(flow)` for each connection. Admission rejects a backend below the policy's required assurance. `Unsupported` means the backend provides no identity; a per-connection `ResolveError` fails identity-scoped policy closed. - -```rust -#[async_trait] -trait IdentitySource: Send + Sync { // the proxy's per-connection resolver - async fn resolve(&self, flow: Flow) -> Result; -} - -enum Identity { Evidence(Evidence), Unsupported } - -struct Evidence { - assurance: Assurance, // ordered for policy; see below - binary_path: PathBuf, - binary_sha256: Option, // None when unavailable - ancestors: Vec, // ancestor binaries, nearest first - cmdline_paths: Vec, // script/interpreter paths from the cmdline -} - -// Ordered for policy: binary-scoped rules require `Observed` or higher, and -// `Claimed` counts as `None` for them. `Attested` is defined narrowly (see -// below); evidence that does not meet that bar is not `Attested`. -enum Assurance { None, Claimed, Observed, Attested } -``` - -`Observed` means the backend identifies the process behind the accepted flow and measures its live executable; a digest is required. `Attested` preserves that flow attribution and verified live-executable digest, then adds fresh cryptographic evidence bound to the boundary and authenticated sandbox claim from a trust root outside the agent's adversary domain. Evidence below those definitions reports a lower assurance. Binary-scoped rules require `Observed` or higher; missing, ambiguous, stale, or failed resolution denies them. - -`Flow` is the backend-issued, opaque token from `MediatedConnection`. The supervisor never interprets it. Procfs lookup, token construction, remote correlation, timeouts, and attestation protocols belong to backend implementations; the interface and fail-closed semantics do not change. - -### Representative topologies - -The security-relevant distinction is whether the supervisor shares the agent's kernel: - -| Class | Kernel-compromise ceiling | Examples | -|---|---|---| -| **Shared host kernel**: supervisor and agent on the host kernel | a host-kernel exploit | in-pod, sidecar, node enforcer, `runc` split-pod | -| **Shared guest kernel**: supervisor and agent share one isolated kernel, a VM guest kernel or a userspace application kernel such as gVisor's; the host is isolated, the supervisor is not | a guest-kernel exploit | single-pod microVM, outer gVisor sandbox | -| **Kernel-separated**: supervisor outside the agent's guest kernel | escaping the agent's guest kernel | split-pod with the agent under a VM or gVisor `runtimeClassName`, future node runtime | - -The first two classes do not contain a kernel-level adversary; sidecar, node, and split-pod placements may still add privilege separation within that ceiling. A single-pod VM or gVisor RuntimeClass places the supervisor inside the agent's guest or application kernel; kernel separation requires the supervisor outside it. In every class, the trusted computing base includes the supervisor, backend, and their control channel. - -The non-normative [selection matrix](./topology-matrix.md) compares concrete placements. +The contract fixes the roles; a topology fixes their placement. The supervisor, network mediation, and backend components may be co-located with the workload, split across components the backend coordinates, or hosted in shared trusted services — the lifecycle, interfaces, and invariants are identical in every arrangement. What containment a placement achieves depends on the kernel relationship between the workload and the trusted components relied on for containment. The non-normative [topology matrix](./topology-matrix.md) catalogs representative placements. ## Implementation plan -Backends arrive incrementally behind one contract: +This RFC defines the contract; implementation lands in three phases: -1. **Land the contract and in-pod adapter.** Add the registry and explicit descriptors behind a rollout gate; preserve current behavior. -2. **Close the in-pod release gates.** Repair egress confirmation and the trusted-init window, then route the supervisor through the adapter and enable it. -3. **Add a sidecar backend (separate design).** Move network-setup privilege out of the agent container; on Kubernetes, map `boundary_ready` to a native sidecar `startupProbe` so agent startup waits for the boundary and closes the `workspace-init` window where sidecar containers are available. -4. **Add a node or split-pod backend (separate design).** Validate establishment outside the supervisor. -5. **Add kernel-separated backends as demand warrants (separate designs).** Each defines its attestation and cross-kernel identity. +1. **Contract.** Add the common types, descriptor handling, registry, and explicit backend selection from deployment configuration. +2. **Co-located backend.** Implement the co-located backend and route agent launch, network mediation, SSH, `exec`, and forwarding through it without changing behavior. +3. **Conformance and enablement.** Add the shared conformance tests — version agreement, lifecycle ordering, process semantics, confirmation, identity failure, execution-environment preservation, and boundary termination — and enable the co-located backend after parity validation. Parity covers the agent, SSH, `exec`, and forwarding paths; enablement also closes the in-pod egress gaps pinned in [codebase-grounding.md](./codebase-grounding.md), which parity alone would preserve. -Drivers emit explicit in-pod descriptors before the supervisor requires them; there is no implicit default. RFC acceptance approves the contract, not the follow-on backends. The RFC becomes `implemented` when the in-pod backend passes its release gates and runs through the contract. - -Shared conformance tests cover version agreement, lifecycle ordering, process semantics, confirmation, identity failure, execution-domain preservation, cleanup, and boundary termination. +Delegated backends remain separate design and implementation work. ## Risks -- **In-pod confirmation is incomplete.** Its nftables ceiling is accept-by-default, covers only TCP and UDP, and disappears when `nft` is absent. Step 2 must establish and confirm effective default-deny before release. -- **Runtime endpoints bypass containment.** Admission must positively constrain reachable runtime endpoints, host namespaces, paths, and devices. -- **The init path runs too early.** `workspace-init` executes image-provided code before egress confinement. The in-pod backend cannot ship until this path uses trusted content or runs behind an established boundary. -- **Contract conformance does not create containment.** Each delegated backend must prove that enforcement and its control channel are outside the workload's reach. -- **Cross-kernel identity is unproven.** Remote correlation is on the connection hot path; timeout or lookup failure denies identity-scoped egress. -- **Adoption spans several entry paths.** Agent launch, SSH, and supervisor sessions must all move behind the contract without changing behavior. +| Risk | Mitigation | +|---|---| +| The Isolation Backend could duplicate compute-driver responsibilities or allow topology-specific behavior to leak back into the supervisor. | Keep the responsibility boundary explicit: the compute driver provisions and deprovisions the topology; the backend binds and operates the active boundary. The same component may implement both roles. | +| Contract conformance could be mistaken for equivalent isolation across topologies. | Treat conformance as behavioral, not as a security-strength rating. Document and validate each topology's actual containment and reject policy it cannot enforce. | +| Shared backend or network-mediation components concentrate privilege and failure impact. | Isolate state, connection attribution, enforcement, and control authority per boundary. Failure of one boundary must not weaken another or enable direct egress. | +| The mandatory contract may exclude otherwise useful but incomplete backends. | Keep network mediation, binary identity, process control, `exec`, and port forwarding mandatory. An incomplete backend does not claim conformance or silently degrade. | +| A future topology may not fit the lifecycle or interfaces. | Keep placement and coordination backend-private. Add versioned contract surface only when a concrete implementation requires new common semantics. | +| A component restart may interrupt boundary operation. | A backend may preserve an active boundary only while it retains one logical supervisor and valid enforcement and runtime-handle semantics; otherwise it terminates the boundary. | ## Alternatives -### Implement a specific topology directly, without the interface +### Keep isolation embedded in the supervisor -OpenShell could wire one delegated topology (the split-pod proposal in #981, for example) straight into the supervisor and the driver. +OpenShell could keep the current in-pod design and add topology-specific supervisor and compute-driver paths as new requirements arise. -This solves one placement but hardwires it. The factory, states, and registry cost less than a supervisor fork for each later placement. Backend transports, `Flow` formats, and attestation remain follow-on design work. +Doing nothing avoids a new interface, but retains privileged boundary construction beside the workload. Implementing each delegated topology as a one-off supervisor change moves that privilege for one placement but accretes topology-specific supervisor behavior. The proposed contract instead keeps one supervisor lifecycle while allowing the topology to change. -### Fold boundary operation into the compute driver +### Extend the compute-driver contract -Establishing the boundary is already a driver concern, so the runtime side could live there too. +The compute driver could own both provisioning and active-boundary operation. -The driver provisions platform resources; the supervisor operates policy, identity, and runtime lifecycle. Putting runtime operations in compute drivers would duplicate the supervisor-facing contract across them. +This is natural for topologies such as MXC, and the same component may implement both responsibilities. The interfaces remain distinct because they serve different callers and lifecycles: the gateway uses the compute driver to provision and deprovision resources, while the supervisor uses the Isolation Backend to operate an active boundary. Combining them would couple runtime policy, identity, network mediation, and process operations to the gateway-facing driver API. -### Select the backend through `SandboxPolicy` +### Define a remote backend service -The backend could be chosen by the workload's `SandboxPolicy` alongside its other isolation settings. +OpenShell could define a gRPC service or plugin ABI for Isolation Backends instead of an in-process Rust contract. -`SandboxPolicy` is workload policy; which backend realizes it is deployment topology. Coupling them would let an untrusted workload influence its own isolation strength and make policy non-portable across deployments, which is why selection sits in admission config instead (see Provisioning the boundary). +That would make process separation and out-of-tree implementations explicit, but it would also standardize transport, authentication, state recovery, and protocol versioning even though those requirements vary by topology. The proposed factory contract supports co-located implementations directly and allows delegated implementations to hide their transport behind the same interface. -### Relax pod semantics to give containers different isolation classes +### Standardize topology and capabilities -A single pod could place its containers in different kernels, so the supervisor and agent share a pod but not a kernel. +The contract could expose common topology roles, placement fields, capability flags, and recovery behavior so the supervisor can compose backend components. -The kernel boundary is selected per pod (`runtimeClassName`), not per container, so this needs a CRI change and a runtime that acts on it, not a backend behind this interface. The supported ways to separate the supervisor's kernel from the agent's are two pods or a future node runtime. - -### Fold this into the proxy egress work (#1511) - -This boundary could be part of #1511, which scopes the proxy pipeline. - -The proxy egress work (#1511) owns the proxy above the boundary. This RFC defines the boundary beneath it, the thing that forces traffic into the proxy. The two are adjacent but distinct concerns. +That would make known deployments explicit, but it would also encode current topology assumptions and introduce capability-dependent supervisor paths. The proposal keeps placement and coordination in the opaque descriptor, requires one baseline contract, and uses the non-normative topology matrix to document representative arrangements. ## Prior art -- **Driver-backed subsystems (CRI/CNI/CSI).** Kubernetes factors runtime, networking, and storage into pluggable driver contracts so the orchestrator drives one interface while implementations vary. RFC 0001 describes OpenShell's other subsystems the same way; this fills in the one it left as a box. +- **Driver-backed subsystems (CRI/CNI/CSI).** Kubernetes factors runtime, networking, and storage into pluggable driver contracts so the orchestrator drives one interface while implementations vary. RFC 0001 describes OpenShell's other subsystems the same way; this RFC specifies the one it left open: isolation. - **Istio privilege placement.** Init-sidecar and node-agent modes demonstrate that network setup can move without changing the policy data path. OpenShell keeps its identity-aware proxy. -- **CRI exec/attach/port-forward.** `exec` and `connect` follow CRI's `Exec` and `PortForward` shape; lifecycle and mediation remain OpenShell-specific. +- **CRI exec/attach/port-forward.** `exec` and `connect` follow CRI's `Exec` and `PortForward` shape; lifecycle and network mediation remain OpenShell-specific. -## Appendix: in-pod migration sketch +## Open questions -The in-pod adapter preserves behavior and privilege while moving existing code behind the states. `attach` creates or joins a closed, unclaimed boundary and returns its control handle; a pooled boundary retains its admitted baseline. `claim` binds the sandbox and policy; `bind` returns mediation, identity, and events; `confirm` checks effective enforcement; and `start_agent` returns `RunningBoundary`. Existing process launch, `exec`, and `connect` paths move behind their runtime interfaces. `BoundaryControl` owns termination and cleanup. - -## Appendix: designing a backend - -A backend-specific design covers its ID and factory, descriptor payload and verification, lifecycle transitions, runtime interfaces, confirmation, policy and resource preservation, provisioning, cleanup, termination, and conformance tests. +None. ## Appendix: codebase grounding -The claims this RFC makes about the current system are verified, with file:line -references, in the supporting file [codebase-grounding.md](./codebase-grounding.md) +The claims this RFC makes about the current system, and the current-system +context behind its design, are verified with file:line references in the +supporting file [codebase-grounding.md](./codebase-grounding.md) (against `origin/main` at commit `a5161d0`, the RFC's parent). diff --git a/rfc/0012-isolation-backend/codebase-grounding.md b/rfc/0012-isolation-backend/codebase-grounding.md index 92e5f8d1a..66810e0c3 100644 --- a/rfc/0012-isolation-backend/codebase-grounding.md +++ b/rfc/0012-isolation-backend/codebase-grounding.md @@ -13,11 +13,11 @@ patterns locate the same code on newer revisions. | Six `setns(CLONE_NEWNET)` call sites the contract's runtime interfaces replace (agent launch, SSH exec and forward, supervisor sessions); plus a `CLONE_NEWNS` at `:393` | `process.rs:589`, `ssh.rs:619`/`:1186`, `supervisor_session.rs:610`, `netns/mod.rs:226`/`:363` (`rg -n "CLONE_NEWNET" crates/openshell-supervisor-process`) | | `nft`-absent fail-open (the invariant bug) | `crates/openshell-supervisor-process/src/netns/mod.rs:264`; logs and returns `Ok(())` at `:272`-`:277` | | In-pod nftables ceiling is accept-by-default and rejects only TCP and UDP, so reading it back does not prove "only the proxy can egress" | `crates/openshell-supervisor-process/src/netns/nft_ruleset.rs:41` (`type filter hook output priority 0; policy accept`), `:43`-`:49` (proxy/loopback/established accept, then `reject` for IPv4 and IPv6 TCP and UDP only; other protocols and raw sockets pass once the host forwards the subnet). `rg -n "policy accept" crates/openshell-supervisor-process/src/netns` | -| Compute driver owns the execution domain (cgroup/resources, security context, device allocation set on the pod by the driver, not the supervisor) | `crates/openshell-driver-kubernetes/src/driver.rs` builds the pod/container spec; `rg -n "securityContext|resources|cdi\.k8s\.io|devices" crates/openshell-driver-kubernetes/src` | +| Compute driver owns the execution domain (cgroup/resources, security context, device allocation set on the pod by the driver, not the supervisor) | `crates/openshell-driver-kubernetes/src/driver.rs` builds the pod/container spec; `rg -n -e securityContext -e resources -e 'cdi\.k8s\.io' -e devices crates/openshell-driver-kubernetes/src` | | VM driver enables forwarding/MASQUERADE (host-forward assumption is load-bearing) | `crates/openshell-driver-vm/src/runtime.rs:417`/`:436` | | No `StartSandbox` RPC (create and start fused; no driver start gate) | `proto/compute_driver.proto` has `CreateSandbox`/`StopSandbox`/`DeleteSandbox` only | | Gateway already speaks exec/session/port-forward; no literal `Attach` | `proto/openshell.proto` (`ExecSandbox`, `ExecSandboxInteractive`, `CreateSshSession`, `ForwardTcp`) | | Agent command via CLI/`SANDBOX_COMMAND`; no admission-bound spec field today; the `sleep infinity` placeholder resolves `sleep` from the agent image's own filesystem | `crates/openshell-sandbox/src/main.rs:331`; K8s driver sets `sleep infinity` at `driver.rs:1886` (`rg -n "sleep infinity" crates/openshell-driver-kubernetes/src`) | -| Init containers: `copy-self` (trusted, the OpenShell binary) and `workspace-init` (runs as root from the agent's own image, so its executables are image-provided); no native sidecars today | `driver.rs:191` (`WORKSPACE_INIT_CONTAINER_NAME`), `:993` (`copy-self` invocation), `:1185` (workspace-init container); `rg -n "restart_policy|workspace-init" crates/openshell-driver-kubernetes/src` | +| Init containers: `copy-self` (trusted, the OpenShell binary) and `workspace-init` (runs as root from the agent's own image, so its executables are image-provided); no native sidecars today | `driver.rs:191` (`WORKSPACE_INIT_CONTAINER_NAME`), `:993` (`copy-self` invocation), `:1185` (workspace-init container); `rg -n -e restart_policy -e workspace-init crates/openshell-driver-kubernetes/src` | | Network policy is OPA per-CONNECT, not the boundary; identity via procfs | `crates/openshell-supervisor-network/src/opa.rs` (`NetworkInput`: `binary_path`/`binary_sha256`/`ancestors`/`cmdline_paths`), `procfs.rs`, glued in `proxy.rs:1611` (`NetworkInput` built in `evaluate_opa_tcp`) | | Static privilege ceiling on every spawned process; OPA never evaluates exec | `process.rs:440` (`ProcessHandle::spawn`), `:603`/`:700` (`drop_privileges` call sites), `:613`/`:705` (sandbox enforcement); SSH reaches the same `enter_netns_and_sandbox` path | diff --git a/rfc/0012-isolation-backend/topology-matrix.md b/rfc/0012-isolation-backend/topology-matrix.md index 384cd6bec..1a5f76dc3 100644 --- a/rfc/0012-isolation-backend/topology-matrix.md +++ b/rfc/0012-isolation-backend/topology-matrix.md @@ -1,108 +1,38 @@ -# Topology selection matrix +# Topology matrix -This non-normative draft maps deployment needs to the placements in -[RFC 0012](./README.md). The RFC defines the contract and kernel-sharing axis; -this file compares placements and suggests when to use them. +This non-normative matrix compares representative mappings of RFC 0012's logical +roles. It records role placement, sharing, and relationship to the workload +kernel; it does not select a deployment or establish conformance. -## Placement catalog +## Representative placements -No placement is universal. The near-term options share the agent's kernel; -stronger options move the supervisor outside it. +| Pattern | Supervisor placement | Backend and network-mediation placement | Workload-kernel relationship | +|---|---|---|---| +| **Co-located/in-pod** | With the workload | Backend runs in the supervisor process; network mediation is co-located | Trusted components share the workload's host, guest, or application kernel, depending on the runtime | +| **Same-pod composite** | With the workload | Backend runs with the supervisor; network mediation may run in a sidecar | Components share the workload's kernel | +| **Delegated backend components** | With the workload | A node or remote helper establishes some controls; network mediation may be co-located or delegated | Depends on which trusted components remain with the workload | +| **Driver-hosted/shared service** | With the compute driver or another trusted service | Backend and network mediation may be co-located with the supervisor; one host may operate many isolated boundaries | Depends on the workload runtime | -**Today-hardened (fallback).** The supervisor remains in the agent pod. This is the simplest path, but isolation privilege remains in the pod and does not meet `restricted`. +## Durable rules -**Sidecar-assisted.** A privileged same-pod sidecar programs network interception. The supervisor remains the proxy and applies filesystem and syscall ceilings. This moves only network-setup capabilities out of the agent container; all containers still share one kernel. A Kubernetes [native sidecar](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/) can map `boundary_ready` to a `startupProbe` that gates agent startup, closing the `workspace-init` window on clusters with sidecar containers available (beta since Kubernetes 1.29, GA in 1.33). Moving the supervisor requires a separate design and an agent-side process shim. +- Every active boundary has one verified descriptor, one trusted + `SandboxContext`, and one logical supervisor. +- Physical processes and listeners may be shared, but lifecycle state, policy, + binary identity, enforcement, and cleanup remain isolated per boundary. +- `Ready` means network mediation is initialized, standing enforcement is + confirmed, and launch-time controls will be in force before untrusted + execution. +- Moving a privileged component does not itself provide kernel separation. -**Node enforcer / daemon set.** A privileged node daemon installs routing and rules during `attach`, so the agent pod needs no network-setup capabilities. The in-pod supervisor still runs the proxy and applies process-time ceilings. NetworkPolicy can add L3/L4 defense in depth but cannot replace identity-aware mediation. This placement adds a privileged node component and does not by itself make the pod capability-free or `restricted`-compatible. +## Kernel relationships -```mermaid -graph LR - subgraph SCPod["sidecar: agent pod"] - direction TB - SCBackend["Isolation Backend (privileged sidecar)"] - subgraph SCAgent["agent + supervisor (network setup delegated)"] - SCWork["workload"] - SCSup["supervisor"] - end - SCBackend -.->|installs egress rules| SCAgent - SCWork ==>|only egress| SCSup - end - SCSup -->|egress| GW1["gateway"] +| Relationship | Meaning | +|---|---| +| **Shared host kernel** | The workload and the trusted components relied on for containment run on the host's kernel. | +| **Shared guest or application kernel** | The workload and those trusted components share one isolated kernel: a VM guest kernel or a userspace application kernel. | +| **Kernel-separated** | The trusted components relied on for containment run outside the workload's kernel. | - NodeD["node daemon (privileged)"] - subgraph NodePod["node enforcer: agent pod (network setup delegated)"] - direction TB - NodeWork["workload"] - NodeSup["supervisor"] - NodeWork ==>|only egress| NodeSup - end - NodeD -.->|installs egress rules| NodePod - NodeSup -->|egress| GW2["gateway"] +## Status - style SCPod fill:#f0f7ff,stroke:#7aaede,color:#00084D - style SCAgent fill:none,stroke:#9575cd,color:#00084D - style SCBackend fill:#fff5f5,stroke:#e57373,color:#00084D - style NodePod fill:none,stroke:#9575cd,color:#00084D - style SCSup fill:#b39ddb,stroke:#9575cd,color:#fff - style NodeSup fill:#b39ddb,stroke:#9575cd,color:#fff - style SCWork fill:#a5d6a7,stroke:#81c784,color:#384D54 - style NodeWork fill:#a5d6a7,stroke:#81c784,color:#384D54 - style NodeD fill:#fff5f5,stroke:#e57373,color:#00084D - style GW1 fill:#a5d6a7,stroke:#81c784,color:#00084D - style GW2 fill:#a5d6a7,stroke:#81c784,color:#00084D -``` - -Splitting the supervisor into a separate pod is one move; whether it separates kernels depends on the agent pod's RuntimeClass. The two variants land in different containment rows (see [RFC 0012's containment table](./README.md#representative-topologies)). - -- **Split-pod, `runc` agent.** Moving the supervisor to another pod separates privilege but not the host kernel. [NetworkPolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) requires an enforcing CNI; pods remain non-isolated for egress until selected by an effective egress policy, and enforcement remains L3/L4 and label-scoped. See #981. -- **Split-pod, VM or gVisor agent.** Running only the agent pod under Kata or gVisor places the supervisor outside the agent's guest or application kernel. A one-pod RuntimeClass does not: it co-locates both processes inside that kernel. -- **Future node runtime.** A node runtime could place supervisor and agent on opposite sides of a kernel boundary without a separate supervisor pod. That backend requires its own design. - -### The design space, and the one cell stock Kubernetes forbids - -Two axes set the space: whether the supervisor shares the agent's kernel, and the pod count or plug point. One combination is impossible on stock Kubernetes. - -| | Supervisor shares the agent's kernel (host or guest) | Supervisor in its own kernel, separate from the agent | -|---|---|---| -| One pod | Today-hardened, sidecar-assisted, and the single-pod outer sandbox (gVisor, Kata, or Firecracker). Per-container `securityContext` varies privilege; the proxy mediates within the pod netns. An outer sandbox isolates the host but runs the supervisor inside the agent's isolated kernel (a VM guest under Kata or Firecracker, gVisor's application kernel under gVisor), so it shares that kernel. | **Not expressible on stock Kubernetes.** The runtime that owns the kernel boundary is selected per pod (`runtimeClassName`), not per container, so two containers of one pod cannot have different kernels. | -| Two pods / node runtime | Node enforcer, or a `runc` split-pod: both share the host kernel. | Split-pod with the agent pod under a VM or gVisor `runtimeClassName` (the supervisor stays in its own pod, outside the agent's guest), or a future node runtime that runs the supervisor outside the agent's kernel. | - -The forbidden cell follows from Kubernetes [RuntimeClass](https://kubernetes.io/docs/concepts/containers/runtime-class/) and the CRI shape: the kubelet selects one runtime handler for `RunPodSandbox`, then creates containers inside that sandbox. Per-container kernels would need a CRI and runtime change, not another backend. In the right column, procfs and netns operations route through the backend. - -### Capabilities, and how they come off - -Removing network setup is only the first step toward an unprivileged agent: - -- A backend removes `NET_ADMIN` and the network use of `SYS_ADMIN` by moving network setup out of the agent's container. -- Removing `SYS_ADMIN` fully needs process and filesystem placement to move too. -- Removing `SYS_PTRACE`/`DAC_READ_SEARCH` requires identity supplied by the backend instead of read in the agent container. - -A sidecar moves network capabilities out of the agent container but not the pod security profile. A node enforcer moves them off the pod. Meeting `restricted` also requires compatible SELinux/MCS labels, seccomp, UID handling, identity, and execution-domain preservation. - -## The dimensions - -Two dimensions choose a placement; data sensitivity raises the posture: - -1. **Infrastructure.** Kernel isolation such as Kata or gVisor enables a kernel-separated placement; otherwise choose among namespace-based options. -2. **Workload.** Arbitrary unreviewed code warrants stronger isolation than a narrow privacy-router workload. -3. **Data sensitivity.** PII or other sensitive data moves the recommendation one step stronger. - -## The matrix - -Rows are infrastructure; columns are what the agent does. Cells name the -recommended topology from RFC 0012. - -| | Agent = privacy-router only (talks to an LLM, low blast radius) | Agent = arbitrary code execution (untrusted code, high blast radius) | -|---|---|---| -| **Has kernel isolation** (nested virt / bare metal; Kata or gVisor available) | **Sidecar-assisted.** A privacy router's blast radius is small, so a kernel boundary's overhead rarely pays for itself against that threat model. (Move to Kata if PII is in play.) | **Kernel-separated.** Put the agent in its own guest kernel, but keep the supervisor outside it: split-pod with the agent pod under Kata/gVisor, or a future node runtime. A one-pod Kata/gVisor RuntimeClass co-locates the supervisor in the same guest kernel. | -| **No kernel isolation** (managed Kubernetes, no nested virt) | **Today-hardened.** Non-root supervisor in the agent pod; the simplest fallback. (Move to sidecar if PII is in play.) | **Sidecar-assisted, then node enforcer.** Sidecar to keep privilege off the agent container; node enforcer to move the network-setup privilege off the pod. The best achievable without a VM. | - -**PII tie-breaker.** If sensitive data is in play, move one cell toward stronger -isolation: a privacy-router deployment adopts the posture its arbitrary-code -neighbor would use, and an arbitrary-code deployment with kernel isolation -available should prefer the kernel-separated path rather than a same-kernel -sidecar. - -## Status and ownership - -This draft is intentionally non-normative. It may evolve into deployment documentation as concrete backends land. +This matrix is non-normative. It illustrates implementations of RFC 0012; it +does not extend the contract.