diff --git a/SPIKE.md b/spike/README.md similarity index 78% rename from SPIKE.md rename to spike/README.md index c39efed18..c2a1ba9c8 100644 --- a/SPIKE.md +++ b/spike/README.md @@ -1,7 +1,18 @@ + + # Node Enforcer Topology Findings Date: 2026-05-29 +## Diagrams + +- [OpenShell isolation architecture](openshell-isolation-diagram.html) +- [Kubernetes isolation tradeoffs](kubernetes-isolation-tradeoffs-diagram.html) +- [Kubernetes node-enforcer option](kubernetes-node-enforcer-options-diagram.html) + ## Summary This branch explored how OpenShell can remove `privileged: true` from sandbox @@ -83,7 +94,8 @@ The current prototype enforcement flow is registration-driven: 3. The node enforcer DaemonSet runs the same supervisor binary with `supervisor_role = "enforcer"`. In that role it does not start an agent - supervisor; it binds `0.0.0.0:17671` and waits for workload registrations. + supervisor; it binds the configured host-network listener and waits for + workload registrations. 4. The workload supervisor starts normally, loads policy, computes the effective network enforcement mode, and registers with the node enforcer before @@ -152,10 +164,9 @@ cluster because enforcement is node-local and pod-local, not gateway-local: The important deployment constraint is that the node enforcer should be treated as shared node infrastructure. A node can only have one process binding the -default host-network listener `0.0.0.0:17671`. If multiple gateway releases each -enable their own node-enforcer DaemonSet on the same nodes with the same listen -address, they will compete for the same port and duplicate the privileged -component. +same host-network listener. If multiple gateway releases each enable their own +node-enforcer DaemonSet on the same nodes with the same listen address, they +will compete for the same port and duplicate the privileged component. For a shared cluster, the expected shape is one node-enforcer DaemonSet per node pool or cluster security boundary, with all participating gateways pointing @@ -172,58 +183,44 @@ registering workload is allowed to ask for enforcement on that pod. ## Validation -Validated against the development Kubernetes cluster using the normal OpenShell -gateway path. GKE was used as a real managed-cluster validation target, but it -should not be part of the implementation criteria. +Validated against the local Kubernetes development cluster using the normal +OpenShell gateway path and the node-enforcer Helm overlay. -The unchanged Kubernetes e2e test passed: +The Kubernetes smoke path passed with the node-enforcer overlay: ```shell -OPENSHELL_GATEWAY_ENDPOINT=http://34.171.165.42 \ - cargo test --manifest-path e2e/rust/Cargo.toml \ - --features e2e-kubernetes \ - --test bypass_detection \ - -- --nocapture +OPENSHELL_E2E_KUBE_CONTEXT=k3d-openshell-dev-tmutch \ +OPENSHELL_E2E_KUBE_EXTRA_VALUES=deploy/helm/openshell/ci/values-node-enforcer.yaml \ +OPENSHELL_E2E_KUBE_BUILD_IMAGES=1 \ +OPENSHELL_E2E_KUBE_TEST=smoke \ + e2e/rust/e2e-kubernetes.sh ``` -Result: - -```text -test bypass_attempt_is_rejected_fast ... ok -1 passed; 0 failed -``` - -The full existing Rust e2e suite also passed unchanged against the Kubernetes -gateway once the local e2e invocation used a named temporary gateway config: +The bypass-detection path also passed with the same overlay: ```shell -XDG_CONFIG_HOME=/private/tmp/openshell-gke/e2e-config \ -OPENSHELL_GATEWAY=gke-e2e \ -OPENSHELL_E2E_DRIVER=kubernetes \ - cargo test --manifest-path e2e/rust/Cargo.toml \ - --features e2e \ - --no-fail-fast \ - -- --nocapture +OPENSHELL_E2E_KUBE_CONTEXT=k3d-openshell-dev-tmutch \ +OPENSHELL_E2E_KUBE_EXTRA_VALUES=deploy/helm/openshell/ci/values-node-enforcer.yaml \ +OPENSHELL_E2E_KUBE_BUILD_IMAGES=1 \ +OPENSHELL_E2E_KUBE_TEST=bypass_detection \ + e2e/rust/e2e-kubernetes.sh ``` -Result: - -```text -all e2e test targets passed -``` - -Important test harness note: the e2e settings test expects the local CLI to be -built with `openshell-core/dev-settings`, which the normal e2e scripts already -do. Direct ad hoc cargo invocations should rebuild `openshell-cli` with that -feature before running the full suite. - -Node-enforcer logs showed the expected action: +Manual validation also created a sandbox through the gateway and confirmed the +node enforcer acted on the sandbox network namespace: ```text Observed sandbox workload registration Reconciling sandbox network enforcement -Installing sandbox network egress enforcement for pod 10.40.1.35 -Sandbox network egress enforcement installed for pod 10.40.1.35 in /proc/309096/ns/net +Installing sandbox network egress enforcement for registered pod +Sandbox network egress enforcement installed for registered pod +``` + +The manual sandbox check then attempted raw direct TCP from the sandbox user +path and observed a fast rejection instead of a hanging bypass attempt. + +```text +direct-connect-exit-code 111 ``` ## Key Debug Finding @@ -239,52 +236,13 @@ The fix was to require that the pod IP is present as a local address in the target namespace: ```text -10.40.1.33 -/32 host LOCAL + + ``` That distinction selected the sandbox pod namespace instead of the host namespace and made the unchanged e2e test pass. -## Envoy Gateway Timeout Finding - -The managed-cluster validation exposed a second, unrelated deployment issue: -Envoy Gateway was terminating long-lived gRPC streams after about 15 seconds. -The symptom was: - -```text -h2 protocol error: error reading a body from connection -``` - -This affected the existing `WatchSandbox`, `ForwardTcp`, and supervisor -`RelayStream` paths used by `sandbox create -- `, upload/create, sync, and -TTY lifecycle tests. The sandbox pods themselves were running and the node -enforcer had installed rules correctly; the failure was at the external gateway -proxy layer. - -The fix belongs to the deployment that installs Envoy Gateway, not to the -OpenShell product Helm chart. Envoy Gateway deployments can attach a -`BackendTrafficPolicy` to the OpenShell `GRPCRoute`: - -```yaml -apiVersion: gateway.envoyproxy.io/v1alpha1 -kind: BackendTrafficPolicy -metadata: - name: openshell-grpc-streams -spec: - targetRefs: - - group: gateway.networking.k8s.io - kind: GRPCRoute - name: openshell - timeout: - http: - requestTimeout: 0s - maxStreamDuration: 0s -``` - -After that policy was applied, Envoy reported it as accepted and the previous -15-second failures passed without test changes. - ## Product Finding This topology shifts the privileged boundary from the sandbox workload to a @@ -297,9 +255,6 @@ That is likely the right direction if the goal is: - OpenShell keeps dynamic network proxy enforcement. - The supervisor remains responsible for the agent lifecycle. - Non-Kubernetes deployments can keep the existing combined supervisor mode. -- External ingress and Gateway API controllers are deployment concerns. If they - impose request or stream duration limits, the deployment must configure them - to preserve OpenShell's long-lived gRPC streams. It is not a claim that the system has no privileged code. On Linux, dynamic network namespace enforcement needs some trusted component with elevated @@ -353,13 +308,9 @@ Known risks: one enforcer DaemonSet per release, which is not safe to enable repeatedly on the same nodes without coordination. - The current rules are coarse: UID 0 is allowed, non-root TCP/UDP is rejected. -- IPv6 namespace lookup exists conceptually but has not been validated in the - managed-cluster path. +- IPv6 namespace lookup exists conceptually but has not been validated. - Observability is useful but should become structured enough for operators to prove what pod/netns/rules were acted on. -- Edge proxies can break otherwise healthy sandboxes if they impose short - request or stream duration timeouts on gRPC. This is deployment-specific and - should be validated wherever OpenShell is exposed through a proxy. ## Hardening Plan @@ -414,8 +365,6 @@ The branch added configuration and code paths for: - Helm `server.sandboxImagePullPolicy` - Kubernetes driver injection of node and pod IP environment variables - Supervisor image packaging with `nftables` -- Deployment-owned Envoy Gateway `BackendTrafficPolicy` for long-lived - OpenShell gRPC streams. The development cluster currently uses: @@ -423,8 +372,6 @@ The development cluster currently uses: - `server.networkEnforcementMode: external-enforcer` - `server.sandboxImagePullPolicy: IfNotPresent` - `nodeEnforcer.enabled: true` -- An Envoy Gateway `BackendTrafficPolicy` with `requestTimeout = 0s` and - `maxStreamDuration = 0s`. ## Recommendation @@ -436,8 +383,3 @@ Do not present it as removing privilege entirely. Present it as moving privilege network enforcement into a narrow, operator-controlled component that can be authenticated, audited, reconciled, and hardened independently from untrusted agent workloads. - -Keep Envoy Gateway and other ingress-controller tuning out of the product chart -unless OpenShell intentionally takes ownership of that controller. The product -requirement is that long-lived gRPC streams must be supported; the deployment -implementation should provide the controller-specific policy. diff --git a/spike/kubernetes-isolation-tradeoffs-diagram.html b/spike/kubernetes-isolation-tradeoffs-diagram.html new file mode 100644 index 000000000..72392f93f --- /dev/null +++ b/spike/kubernetes-isolation-tradeoffs-diagram.html @@ -0,0 +1,434 @@ + + + + + + + + OpenShell Kubernetes Isolation Tradeoffs + + + +
+
+
+

Kubernetes Isolation Tradeoffs

+

+ Current OpenShell keeps the supervisor and agent in one Kubernetes sandbox pod. The split-pod proposals move untrusted agent execution into a separate pod and use platform egress controls, while the Isolation Backend proposal turns the boundary machinery into an explicit contract. +

+
+
current vs proposed states
+
+ +
+ + OpenShell Kubernetes isolation tradeoffs + Comparison of current in-pod isolation, proposed split-pod isolation, and an isolation backend abstraction for OpenShell Kubernetes sandboxes. + + + + + + + + + + + + + + + + + + + + + + + + + + + + A. Current Kubernetes State + single sandbox pod, supervisor builds the inner boundary + + + B. Split-Pod Proposal + trusted supervisor pod, untrusted agent pod, CNI egress boundary + + + C. Isolation Backend Interface + common supervisor contract, backend-specific boundary realization + + + + Sandbox Pod + + + openshell-sandbox supervisor + root prelude, proxy, OPA, credentials + SSH relay, logs, process spawn + requires elevated pod capabilities + + + Inner network namespace + veth route forces egress to proxy + nftables rejects direct TCP/UDP bypass + bypass monitor records attempts + + + Restricted agent process + sandbox user, Landlock, seccomp + binary identity via /proc + + + spawn + + enter namespace + + + Current tradeoffs + + Strong in-pod egress forcing independent of CNI + + Per-binary network policy input still available + - Supervisor privilege sits in same pod as agent + - Blocks restricted-v2 / hardened managed clusters + - Agent and supervisor logs share the workload stream + + + + Supervisor Pod + + + Proxy + OPA + L7 policy gate + + + Credentials + never in agent pod + + + Agent Pod + + + Untrusted image + non-root, no caps + gVisor or Kata + + + Sideload binary + optional SSH/Landlock + + + Kubernetes NetworkPolicy + allow: agent pod to supervisor:3128 + allow: DNS when configured + deny: all other direct egress at CNI layer + + + RuntimeClass + gVisor userspace kernel or Kata VM boundary + + + HTTP_PROXY + + non-cooperative traffic + + + + Split-pod tradeoffs + + Agent pod can run non-root with zero capabilities + + gVisor/Kata reduces host-kernel blast radius + + Separate log streams for audit and agent output + - Per-binary destination restrictions are lost cross-pod + - Enforcement now depends on CNI NetworkPolicy support + + + + Supervisor stays policy authority + proxy, OPA, credentials, audit, inference routing + drives one runtime contract instead of inline topology code + + + Isolation Backend contract + realize boundary, return verifiable handle + fail closed if boundary is not ready + operator-selected, not workload-selected + + + In-pod + netns + nft + status quo path + + + Delegated + split pod + CNI boundary + + + VM / Node + microVM + node agent + + + Two invariants + no unguarded workload egress before verification + no workload execution before boundary ready + + + + + + + + + Abstraction tradeoffs + + Separates policy authority from boundary machinery + + Lets split-pod, VM, or node-agent backends share a contract + + Enables in-pod refactor first, delegated later + - Requires verifiable handoff and fail-closed semantics + - Scope is unsettled: network only vs fs/syscall/process too + + + + Decision Map + The diagrams point to different places for the hard boundary. That changes who must be trusted and which controls are enforceable. + + + Hard egress boundary + current: netns + nftables inside pod + split-pod: CNI NetworkPolicy outside agent pod + backend: selected per environment, verified by contract + + + Kernel isolation + current: runc + seccomp + Landlock + split-pod: gVisor userspace kernel or Kata VM + backend: can represent in-pod, VM, or delegated runtime + + + L7, credentials, inference + current: local supervisor proxy + split-pod: supervisor pod proxy + backend: remains with supervisor policy authority + + + Known loss / risk + current: privileged setup near untrusted workload + split-pod: per-binary policy and deny logs become weaker + backend: verification and versioning become first-class + + + Best fit + local dev, k3d/k3s, environments allowing capabilities + enterprise clusters, OpenShift/Gardener-style restrictions + long-term architecture that lets both shapes coexist + + + + Sources: current code/docs; #899 restricted SCC; #981 split supervisor/agent pod; #1305 proxy/runtime decomposition; #1650 process/network split PR; #1703 external compute driver PR; #1737 Isolation Backend interface. + +
+ +
+
+

Current State

+

The Kubernetes driver creates one Sandbox CR whose pod runs the agent container with supervisor sideloading and elevated capabilities so the supervisor can create the inner network namespace.

+
+
+

Split Pod

+

The proposal moves the proxy and policy authority into a trusted supervisor pod. The agent pod becomes untrusted, non-root, no-capabilities, and optionally gVisor or Kata isolated.

+
+
+

NetworkPolicy

+

Today Helm renders an ingress-only sandbox SSH NetworkPolicy. The split-pod proposal adds egress NetworkPolicy so agent pods can reach only the supervisor proxy and DNS.

+
+
+

Backend Interface

+

The interface proposal keeps the supervisor as policy authority while letting in-pod, split-pod, VM, or node-agent backends realize the isolation boundary with explicit verification.

+
+
+ +
+ Sources used: #899, + #981, + #1305, + PR #1650, + PR #1703, + #1737, + plus local files architecture/compute-runtimes.md, + deploy/helm/openshell/templates/networkpolicy.yaml, and + crates/openshell-driver-kubernetes/src/driver.rs. +
+
+ + diff --git a/spike/kubernetes-node-enforcer-options-diagram.html b/spike/kubernetes-node-enforcer-options-diagram.html new file mode 100644 index 000000000..867a66f05 --- /dev/null +++ b/spike/kubernetes-node-enforcer-options-diagram.html @@ -0,0 +1,412 @@ + + + + + + + + OpenShell Kubernetes Node Enforcer Option + + + +
+
+
+

Kubernetes Node Enforcer Option

+

+ The node-enforcer branch adds a fourth Kubernetes isolation shape: keep the workload supervisor and proxy in the sandbox pod, but move privileged pod-network-namespace enforcement to an operator-controlled DaemonSet on each node. +

+
+
branch: node-enforcer-poc/tmutch
+
+ +
+ + OpenShell Kubernetes node-enforcer option + Diagram of the node-enforcer topology and comparison to supervisor-netns, soft-proxy, split-pod, and isolation backend options. + + + + + + + + + + + + + + + + + + + + + + + + + + + + Option 4: Node-Enforcer Topology + workload supervisor remains in the sandbox pod; privileged netns/nft enforcement moves to a node-local DaemonSet + + + + Gateway + creates Sandbox CR + injects role + enforcement mode + + + Kubernetes Driver + supervisor_role=workload + network=external-enforcer + + + + Kubernetes node + + + node-enforcer DaemonSet + same supervisor image, role=enforcer + hostNetwork + hostPID + privileged + listens on host-network endpoint + + + Namespace lookup + find pod netns by pod IP in /proc + + + Install nft table + setns(CLONE_NEWNET) + nft -f + + + Sandbox pod network namespace + + + Workload supervisor + policy loading, proxy, SSH, logs, agent lifecycle + registers with node enforcer before spawning agent + + + Local proxy + OPA + L7 + creds + + + Agent child + sandbox user + + + openshell_external_enforcer table + + + External services + allowed only through proxy + + + Gateway session + config, relay, logs + + + create sandbox + + inject env + + + HTTP registration + pod IP + sandbox ID + + + install pod-netns nft rules + + + HTTP_PROXY + + UID 0 proxy egress + + non-root direct socket rejected + + gateway callback + + + What moved out of the workload pod? + + netns discovery and nftables installation are node-side infrastructure duties + + workload supervisor keeps policy/proxy/credentials close to the agent + + + Prototype hardening required: authenticate registrations, verify pod ownership, reconcile cleanup/stale state, avoid per-release DaemonSet port conflicts. + + + + Comparison Against Other Kubernetes Options + The options differ mostly by where the hard network boundary lives and whether the workload pod needs elevated privileges. + + + Option + Hard egress boundary + Privileged component + What remains local to agent pod + Main tradeoff + + + Supervisor-netns + inner netns + veth + nft + sandbox workload supervisor + all policy, proxy, fs, process, netns + strong but requires elevated sandbox pod + + + Soft-proxy + none for direct sockets + none for network setup + proxy, policy, lifecycle, logs + deployable but bypass is not kernel-blocked + + + Split-pod + CNI NetworkPolicy + trusted supervisor pod / platform + agent only; optional sideload SSH/Landlock + loses trusted per-binary identity cross-pod + + + Node-enforcer + pod netns nft table installed by DaemonSet + node enforcer only + proxy, OPA, creds, SSH, agent lifecycle + keeps proxy local; moves network privilege out + + + Isolation Backend + selected backend: netns, CNI, node, VM + backend-specific + supervisor as policy authority + needs verifiable handoff and fail-closed contract + + + + How Node-Enforcer Fits the Isolation Backend Abstraction + + + Supervisor contract + policy authority, proxy, audit + register before agent exec + + + Backend handle + pod IP + sandbox ID + node-local enforcer endpoint + + + Backend mechanism + hostPID /proc scan + setns + nft ruleset + + + Required invariant + no agent execution before + boundary is installed/verified + + + + + + + Open questions + authn/authz + cleanup + ownership checks + + Sources: node-enforcer-poc/tmutch branch files architecture/plans/node-enforcer-topology-findings.md, crates/openshell-sandbox/src/enforcer.rs, crates/openshell-sandbox/src/lib.rs, crates/openshell-driver-kubernetes/src/driver.rs, deploy/helm/openshell/templates/node-enforcer.yaml, docs/reference/sandbox-compute-drivers.mdx, plus proposal context from #981 and #1737. + +
+ +
+
+

What It Preserves

+

The workload supervisor still owns proxy, OPA, credential resolution, inference routing, SSH, logs, settings reloads, and agent lifecycle inside the sandbox pod.

+
+
+

What It Moves

+

The node enforcer owns pod network namespace discovery and coarse nftables egress enforcement, moving that privilege out of the sandbox workload pod.

+
+
+

What It Does Not Do

+

It does not split the agent into a separate pod and does not use Kubernetes NetworkPolicy as the hard boundary. Dynamic endpoint and L7 policy still live in the proxy.

+
+
+

Hardening Work

+

The POC needs authenticated registration, pod ownership verification, cleanup/reconciliation for stale namespaces, and shared-cluster deployment rules.

+
+
+ + +
+ + diff --git a/spike/openshell-isolation-diagram.html b/spike/openshell-isolation-diagram.html new file mode 100644 index 000000000..efdb2d4a0 --- /dev/null +++ b/spike/openshell-isolation-diagram.html @@ -0,0 +1,559 @@ + + + + + + + + OpenShell Isolation Architecture + + + +
+
+
+
+ +

OpenShell Networking + Filesystem Isolation

+
+

+ Gateway policy delivery stays separate from sandbox-local enforcement. The supervisor prepares static controls, launches the agent as an unprivileged child, and forces observable egress through the policy proxy. +

+
+
+ Standalone HTML/SVG
+ Generated from repo docs + sandbox code +
+
+ +
+ + OpenShell isolation architecture + A three-panel architecture diagram showing gateway and sandbox boundaries, network egress enforcement, and filesystem startup isolation. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 1. Runtime Boundary + Gateway delivers desired state. Sandbox supervisor owns runtime enforcement. + + + 2. Network Egress Enforcement + Ordinary traffic is forced through the proxy, evaluated, and either forwarded, denied, logged, or routed to inference.local. + + + 3. Filesystem + Process Startup + Static controls are prepared before exec and normally require sandbox recreation to change. + + + + + CLI / SDK / TUI + user-facing API clients + + + + Gateway Control Plane + sandbox state, policy revisions + settings, credentials, inference routes + + + + Compute Runtime + Docker, Podman, K8s, VM + + + Sandbox workload + + + + Supervisor + root prelude, policy load, config polling + proxy, SSH relay, log push + + + + Restricted Agent Child + sandbox user + group + Landlock, seccomp, netns active + + + + OPA + policy engine + + + + Proxy + egress gate + + + + Relay + connect/exec/fs + + + gRPC / HTTP + + + launch request + + + provisions workload + + + outbound supervisor session + config refresh, logs, relays + + + spawn + restrict + + + + + + + + + Agent Process + HTTP(S), git, model SDKs + env: http_proxy / HTTPS_PROXY + + + + Bypass Guards + netns + veth route + nft rejects non-proxy TCP/UDP + + + + Local CONNECT Proxy + proxy endpoint inside netns + terminates selected TLS/L7 + + + + Binary Identity + path, sha256, ancestors + + + + SSRF Hard Blocks + internal IPs unless allowed + + + + OPA L4 Decision + host, port, binary, policy + + + + TLS / L7 Inspection + REST, GraphQL, WebSocket + method, path, operation, fields + + + + External Services + allowed upstream traffic + credentials resolved only in proxy + + + + inference.local + special route, not OPA network + + + + openshell-router + configured model backends + + + + Deny / Audit / Log + OCSF + structured 403 + + + + Policy Proposals + L4 denial summaries to gateway + + + ordinary egress + + bypass attempt + + + + + + allowed + protocol inspect + + forward + + special host + + + deny wins + + L4 DenialEvent + + + + + Policy YAML + filesystem_policy + read_only + read_write + + + + Supervisor Prelude + load + validate policy + enrich baseline paths + workdir and runtime material + + + + Landlock Prepare + open PathFds as root + build ruleset + best_effort or hard_requirement + + + + pre_exec Child + enter netns/mount ns + drop to sandbox user + no supervisor identity socket + + + + Enforce Controls + restrict_self() + seccomp filter + raw socket and privilege paths blocked + + + + Exec Agent + restricted runtime + undeclared paths denied + + + + Read-only paths + /usr, /lib, /etc, /app... + + + + Read-write paths + /sandbox, /tmp, workdir... + + + + Runtime enrichment + proxy support, GPU devices + + + + Blocked by default + undeclared filesystem paths + + + + + + + + + + + + + + + Legend + + client / protocol inspection + + gateway or supervisor control + + policy engine / policy data + + security enforcement / deny path + +
+ +
+
+
+ +

Network Isolation

+
+
    +
  • Traffic is configured for proxy mode in sandbox policy conversion.
    • +
  • The child receives proxy environment variables before exec.
    • +
  • Network namespace and nftables reject direct TCP/UDP bypass attempts.
    • +
+
+ +
+
+ +

Filesystem Isolation

+
+
    +
  • Landlock PathFds are opened while the supervisor still has root privileges.
    • +
  • The child drops to the sandbox user before Landlock and seccomp enforcement.
    • +
  • Read-only, read-write, workdir, and enriched runtime paths define the visible filesystem.
    • +
+
+ +
+
+ +

Dynamic Policy

+
+
    +
  • Network policy can reload for new connections or safely parsed HTTP requests.
    • +
  • Filesystem and process controls are startup-time controls.
    • +
  • L4 denials can become pending gateway policy proposals.
    • +
+
+
+ + +
+ +