OpenShell/docs/kubernetes/topology.mdx

---
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
title: "Kubernetes Sandbox Topology"
sidebar-title: "Topology"
description: "Choose between combined and sidecar supervisor topology for Kubernetes sandbox pods."
keywords: "Generative AI, Cybersecurity, Kubernetes, Sandboxing, Sidecar, Network Policy, RuntimeClass"
position: 2
---

Kubernetes sandbox pods can run the OpenShell supervisor in `combined` or
`sidecar` topology. Choose the topology based on which controls you need inside
the pod and how much privilege your cluster allows on the agent container.

## Choose a Topology

The default `combined` topology preserves the full OpenShell enforcement model.
Use `sidecar` only when you accept network-focused enforcement in exchange for a
lower-privilege agent container.

| Topology | Use when | Main tradeoff |
|---|---|---|
| `combined` | You need OpenShell network, filesystem, and process controls in the sandbox workload. | The agent container carries the Linux capabilities the supervisor needs. |
| `sidecar` | You need the agent container to run as non-root without added Linux capabilities, and network policy is the primary control. | Privilege-dropping and supervisor mount isolation do not run in the agent container. |

## Privilege Model

The long-running container permissions differ by topology:

| Topology | Pod or container | UID/GID | Privilege escalation | Capabilities | Result |
|---|---|---|---|---|---|
| `combined` | Agent container, which also runs the supervisor | Not forced by topology | Not explicitly disabled by the driver | Adds `SYS_ADMIN`, `NET_ADMIN`, `SYS_PTRACE`, and `SYSLOG`; adds `SETUID`, `SETGID`, and `DAC_READ_SEARCH` when user namespaces are enabled | Full supervisor controls run in the agent container. |
| `sidecar` | Agent container, process-only supervisor (`network-only`) | `sandbox_uid:sandbox_gid` | `false` | Drops `ALL` | Agent and workload run without added Linux capabilities. |
| `sidecar` | Network supervisor sidecar, binary-aware mode (default) | `0:sandbox_gid` | `false` | Drops `ALL`; adds `SYS_PTRACE` and `DAC_READ_SEARCH` | Root sidecar inspects cross-UID workload `/proc` entries. The nftables fence exempts UID 0, so do not inject other root containers into these pods. |
| `sidecar` | Network supervisor sidecar, endpoint/L7-only mode | `proxyUid:sandbox_gid` | `false` | Drops `ALL` | Non-root sidecar enforces endpoint and L7 policy without matching `policy.binaries`. |

Short-lived setup containers still have the permissions needed to prepare the
pod:

| Topology | Setup container | UID/GID | Privilege escalation | Capabilities | Purpose |
|---|---|---|---|---|---|
| `combined` | Supervisor install init container | `0` | Not set | Not set | Copies the supervisor binary into the agent container volume. |
| `sidecar` | Network init container | `0` | `false` | Drops `ALL`; adds `NET_ADMIN`, `NET_RAW`, `CHOWN`, and `FOWNER` | Installs pod-local nftables rules and prepares shared sidecar state. |

## Combined Topology

Combined topology is the original Kubernetes mode and remains the default. The
agent container starts the OpenShell supervisor, and the supervisor launches the
workload after applying sandbox setup.

```mermaid
flowchart TB
  Sandbox["agents.x-k8s.io Sandbox"]

  subgraph Pod["Sandbox pod"]
    subgraph Agent["agent container"]
      Supervisor["OpenShell supervisor<br/>network + process + filesystem"]
      Workload["Agent workload"]
    end
  end

  Gateway["OpenShell Gateway"]
  External["External services"]

  Sandbox --> Pod
  Supervisor --> Workload
  Supervisor -->|"gateway callback / SSH relay"| Gateway
  Supervisor -->|"policy-enforced egress"| External
```

Combined topology keeps these controls in one supervisor path:

- Network endpoint and L7 policy enforcement.
- Filesystem policy enforcement.
- Process and binary identity checks.
- Privilege drop into the sandbox user.
- Gateway relay, SSH sessions, exec, and file sync.

Because the supervisor performs network namespace setup and process/filesystem
controls from the agent container, Kubernetes grants that container elevated
Linux capabilities. Use this mode when you need the complete OpenShell sandbox
contract and your cluster policy permits those capabilities.

## Sidecar Topology

Sidecar topology splits the supervisor into a network sidecar and a
low-privilege process supervisor in the agent container.

```mermaid
flowchart TB
  Sandbox["agents.x-k8s.io Sandbox"]

  subgraph Pod["Sandbox pod"]
    Init["network init container<br/>root setup capabilities"]
    State["shared state + TLS volumes"]
    NetNS["pod network namespace"]

    subgraph Agent["agent container"]
      ProcessSupervisor["process supervisor<br/>network-only"]
      Workload["Agent workload"]
    end

    NetworkSidecar["network supervisor sidecar<br/>UID 0 by default"]
    SshEndpoint["abstract SSH relay socket<br/>peer-PID authenticated"]
  end

  Gateway["OpenShell Gateway"]
  External["External services"]

  Sandbox --> Pod
  Init -->|"installs nftables rules"| NetNS
  ProcessSupervisor --> Workload
  Workload -->|"egress redirected on loopback"| NetworkSidecar
  NetworkSidecar -->|"gateway session + relays"| Gateway
  NetworkSidecar -->|"policy-enforced egress"| External
  NetworkSidecar -->|"control socket + proxy TLS"| State
  ProcessSupervisor -->|"bootstrap + updates"| State
  ProcessSupervisor --> SshEndpoint
  NetworkSidecar -->|"SSH relay"| SshEndpoint
  NetworkSidecar --- State
```

The pod contains these OpenShell-managed pieces:

| Component | Runs as | Purpose |
|---|---|---|
| Network init container | Root with setup capabilities | Installs pod-level nftables rules and prepares shared sidecar state. |
| Network sidecar | UID 0 by default; `supervisor.sidecar.proxyUid` when binary-aware policy is disabled | Runs the proxy, enforces network policy, owns gateway authentication and the gateway session, and serves local policy/provider state over the sidecar control socket. |
| Agent container | Resolved sandbox UID/GID | Runs the process supervisor and launches the user workload. |

In this topology, the agent container defaults to `runAsNonRoot: true`,
`allowPrivilegeEscalation: false`, and `capabilities.drop: ["ALL"]`. The
default binary-aware network sidecar runs as UID 0, drops default Linux
capabilities, and adds `SYS_PTRACE` plus `DAC_READ_SEARCH` for cross-UID workload
process identity resolution. Setting
`supervisor.sidecar.processBinaryAwareNetworkPolicy=false` runs the sidecar as
the configured non-root `proxyUid`, omits both capabilities, and downgrades
network policy to endpoint/L7 enforcement without binary matching. The root
init container keeps the setup capabilities needed to configure pod networking.

Sidecar mode preserves gateway session behavior, including SSH connectivity,
because the network sidecar owns the gateway session and bridges relay requests
to a Linux abstract SSH socket owned by the process supervisor. The relay
verifies the socket peer PID against the authenticated control connection, so
the workload cannot replace the relay endpoint. The agent container does not get a
gateway endpoint, gateway TLS material, or the sandbox bootstrap token in the
default sidecar path.

<Warning>
Sidecar mode runs the process supervisor in `network-only` mode. OpenShell still
enforces network endpoint and L7 policy through the sidecar, and the process
supervisor applies Landlock filesystem policy and child seccomp filters where
the kernel/runtime supports them. The process supervisor does not perform
root-to-sandbox privilege dropping because Kubernetes starts the container as
the sandbox UID/GID, and it does not perform supervisor identity mount
isolation because gateway credentials are not mounted into the agent container.
Sidecar pods use `shareProcessNamespace: true` so the network sidecar can
resolve workload process and binary identity through `/proc/<entrypoint-pid>`.
</Warning>

## Credential Exposure

Sidecar topology keeps gateway credentials in the network sidecar. The agent
container does not mount the projected ServiceAccount token used for sandbox
token bootstrap, does not mount the sandbox client TLS secret, and does not get
gateway callback environment variables.

The network sidecar serves the policy and workload-facing provider environment
over a Unix control socket in the shared sidecar state volume. Before launching
the workload, the process supervisor establishes the only accepted connection.
The sidecar validates its UID, GID, and PID with peer credentials, unlinks the
listener, derives the SSH target from trusted configuration, and rejects later
clients. The connection receives bootstrap state and provider-environment
updates after settings polls. If it closes, the network sidecar exits so
Kubernetes recreates the one-client bootstrap listener, and the process
supervisor exits so Kubernetes terminates the workload and restarts the agent
container. This symmetric failure behavior prevents a surviving workload from
claiming the new control listener after an isolated sidecar restart. Future
child processes can see refreshed provider env without giving the agent
container gateway authentication material. This does not mutate the environment
of the already-running workload entrypoint. Use `combined` topology when you
need the full single-supervisor enforcement path; use additional runtime
isolation when you need a stronger container boundary around sidecar workloads.

## RuntimeClass Isolation

Sidecar topology has been validated with Kata Containers. It does not currently
support gVisor because sidecar mode requires pod-local nftables setup, which
gVisor does not provide to the init container. A supported sandboxed runtime
strengthens the container boundary while OpenShell focuses on network policy
enforcement from the sidecar.

Runtime classes do not re-enable the OpenShell privilege-drop or supervisor
mount-isolation controls that sidecar mode relaxes. Use them as an additional
workload boundary, not as a replacement for the combined topology's full
supervisor controls.

You can set a default runtime class in the Kubernetes driver configuration or
override it per sandbox with driver config:

```shell
openshell sandbox create \
  --driver-config-json '{"kubernetes":{"pod":{"runtime_class_name":"kata-containers"}}}' \
  -- claude
```

## Enable Sidecar Mode

For direct gateway TOML configuration, set the Kubernetes driver fields:

```toml
[openshell.drivers.kubernetes]
topology = "sidecar"

[openshell.drivers.kubernetes.sidecar]
proxy_uid = 1337
```

`proxy_uid` configures only the relaxed endpoint/L7-only sidecar. It must be a
non-root UID and must not match the sandbox UID. The default binary-aware mode
runs the sidecar as UID 0 instead. The network init container exempts the
effective sidecar UID from proxy redirection so the sidecar can reach the
gateway.

When the Helm chart renders `gateway.toml`, set the equivalent chart values:

```yaml
supervisor:
  topology: sidecar
  sidecar:
    proxyUid: 1337
    processBinaryAwareNetworkPolicy: true
```

Leave `topology` unset, or set it to `combined`, to keep the original
single-container supervisor path. For Helm installs, leave
`supervisor.topology` unset or set it to `combined`.

Set `supervisor.sidecar.processBinaryAwareNetworkPolicy=false` only when you
accept downgrading sidecar network policy to endpoint/L7 enforcement without
matching `policy.binaries`. This changes the sidecar from UID 0 to `proxyUid`
and removes its `SYS_PTRACE` and `DAC_READ_SEARCH` capabilities, which are used
for cross-UID `/proc` inspection.

## Next Steps

- To install OpenShell on Kubernetes, refer to [Setup](/kubernetes/setup).
- To configure gateway authentication, refer to [Access Control](/kubernetes/access-control).
- To review the driver fields, refer to [Gateway Configuration File](/reference/gateway-config).