trending/OpenShell
1
0
Fork 0
mirror of https://github.com/NVIDIA/OpenShell.git synced 2026-10-25 06:42:52 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat(supervisor): port streamed configuration to RFC 0012

Browse Source 
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
This commit is contained in:
Piotr Mlocek
2026-09-16 11:09:31 -07:00
parent 940487c5be 43d7b2816c
commit 9be177424b
428 changed files with 65234 additions and 33939 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/build-openshell-mxc-windows/SKILL.md
+15 -9
View File
@@ -115,7 +115,7 @@ The lane targets a Windows host with Visual Studio Build Tools and rustup.
| Visual C++ ARM64 tools | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Tools.ARM64 -property installationPath` | Required for native ARM64 check, build, and tests and for x64-to-ARM64 check/build. Tests always require a native runner. |
| Visual C++ ARM64 Spectre-mitigated libraries | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Runtimes.ARM64.Spectre -property installationPath` | Required by `regorus` through `msvc_spectre_libs`; the build fails when the selected MSVC toolset lacks `lib\spectre\arm64`. |
| Visual C++ Clang tools | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Llvm.Clang -property installationPath` | Provides host-native `libclang.dll` for `bindgen` and `clang-cl.exe` for ARM64 crypto dependencies such as `aws-lc-sys`. On ARM64, the wrapper uses `VC\Tools\Llvm\Arm64\bin`. |
| Visual C++ CMake tools | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.CMake.Project -property installationPath` | Provides CMake and Ninja for bundled Z3 and other native dependencies. The x64-to-ARM64 path adds Ninja to `PATH`; Z3 uses MSVC's Visual Studio generator. |
| Visual C++ CMake tools | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.CMake.Project -property installationPath` | Provides CMake and Ninja for native dependencies. The x64-to-ARM64 path adds Ninja to `PATH`; Z3 uses an architecture-specific prebuilt release. |
| Windows SDK | `where.exe rc.exe` from a Developer PowerShell | Install an SDK containing target libraries and ARM64 tools. |
| Rust via rustup | `rustc --version` | Add each target being validated: `x86_64-pc-windows-msvc` and/or `aarch64-pc-windows-msvc`. The wrapper also adds the selected target. |
| mise | `mise --version` | Used as a task runner only. |
@@ -135,6 +135,8 @@ from this skill.
| `CARGO_TARGET_DIR` | `target` under repo root | Override Cargo output location. Use a short absolute path when x64-to-ARM64 builds approach Windows path-length limits. |
| `Z3_LIBRARY_PATH_OVERRIDE` | unset | Directory containing an x64 system `libz3.lib`; not valid for ARM64. |
| `Z3_SYS_Z3_HEADER` | unset | Full `z3.h` path required with a system Z3 library. |
| `Z3_SYS_Z3_VERSION` | `4.16.0` | Pinned official prebuilt Z3 release selected by the wrapper. |
| `READ_ONLY_GITHUB_TOKEN` | unset | Optional token for the Z3 release lookup; GitHub Actions supplies `github.token`. |
| `RUSTC_WRAPPER` | inherited | The wrapper resolves an available command to an absolute path. If it is unavailable, the wrapper warns and continues without compiler caching. |
Legacy fork variables such as `OPENSHELL_UPSTREAM`,
@@ -203,8 +205,7 @@ jobs in the current mirror push run, or push a new mirrored commit. The binaries
The ARM64 check/build steps in this x64-host contract are cross-builds. The
wrapper discovers and adds host-native LLVM and Ninja to `PATH`, requires the
ARM64 compiler and Spectre-mitigated libraries, lets ARM64 crypto crates select
`clang-cl`, and builds bundled Z3 with native MSVC `cl.exe` and the Visual
Studio generator.
`clang-cl`, and downloads the official prebuilt ARM64 Z3 static library.
On ARM64 hosts, validate the native ARM64 check, build, and test path. The
wrapper rejects test targets that do not match the host architecture, so x64
@@ -213,6 +214,10 @@ compatibility under emulation is not part of these tasks. The aggregate
commands above on an ARM64 host.
The repository-wide `mise run pre-commit` task is also supported on Windows.
Run `rust:lockfiles:check`, `sdk:ts:ci`, `go:ci`, and `test:e2e-parity` through
the Windows-aware tasks when validating those surfaces. Do not count the Go
Windows ARM64 race-detector exclusion or POSIX permission-bit skips as security
coverage. SDK test dependencies must remain at their lockfile versions.
Its Rust check, Clippy, and test dependencies enter the same MSVC environment
for the native host target and use an inherited compiler wrapper when it is
available. Linux glibc
@@ -311,12 +316,13 @@ Useful log files:
| `test-x86_64-pc-windows-msvc-unsupported-*.log` | Focused unsupported-driver contract output. |
| `test-aarch64-pc-windows-msvc-unsupported-*.log` | Focused native ARM64 contract output. |
The first check builds bundled Z3 from source through `z3-sys`. Cargo stores the
native build output in its target tree, so the Windows target cache reuses it.
The resulting release executables do not require `libz3.dll`. The artifact
report computes SHA256 through .NET directly and does not rely on the
`Get-FileHash` module being available inside the mise-launched Windows
PowerShell process.
The first check downloads the pinned official Z3 archive for the target
architecture through `z3-sys`. GitHub Actions authenticates the lookup with its
read-only workflow token; local users can set `READ_ONLY_GITHUB_TOKEN` if an
unauthenticated lookup is rate-limited. Cargo stores the extracted library in
its target tree, so the Windows target cache reuses it. The artifact report
computes SHA256 through .NET directly and does not rely on the `Get-FileHash`
module being available inside the mise-launched Windows PowerShell process.
## Common Fix Patterns
.agents/skills/build-openshell-mxc-windows/reference.md
+12 -10
View File
@@ -68,11 +68,11 @@ file.
For ARM64, verify the Visual Studio instance contains the ARM64 MSVC tools,
ARM64 Spectre-mitigated libraries, Clang tools, CMake tools, and a Windows SDK.
Clang supplies host-native `libclang.dll` for `bindgen` and `clang-cl.exe` for
ARM64 crypto dependencies such as `aws-lc-sys`. Native builds use the normal
bundled-Z3 CMake path. An x64-to-ARM64 check/build discovers and adds
host-native Ninja to `PATH`, builds bundled Z3 with native MSVC `cl.exe` and
the Visual Studio generator, and lets the crypto crates select `clang-cl`. Use
a short `CARGO_TARGET_DIR` if Windows path-length limits are reached.
ARM64 crypto dependencies such as `aws-lc-sys`. Native and
x64-to-ARM64 builds use the official prebuilt Z3 4.16.0 static library for the
target architecture. An x64-to-ARM64 check/build discovers and adds host-native
Ninja to `PATH`, while the crypto crates select `clang-cl`. Use a short
`CARGO_TARGET_DIR` if Windows path-length limits are reached.
## Unsupported Driver Rules
@@ -112,16 +112,18 @@ top-level workspace targets for check/test:
--exclude openshell-driver-vault
--exclude openshell-driver-vm
--exclude openshell-sandbox
--exclude openshell-supervisor-network
--exclude openshell-supervisor
--exclude openshell-supervisor-process
--exclude openshell-vfio
```
The gateway keeps platform configuration and unsupported-operation contracts
without depending on the Docker, Kubernetes, Podman, sandbox supervisor,
process supervisor, VM, or VFIO runtime crates. The Kubernetes Secrets and
Vault libraries still compile as gateway dependencies; only their standalone
Unix-socket binaries and package-level tests are excluded as top-level targets.
without depending on the Docker, Kubernetes, Podman, sandbox runtime,
standalone supervisor, supervisor process runtime, VM, or VFIO crates. The MXC
driver does depend on the cross-platform supervisor network library for its host
egress proxy. The Kubernetes Secrets and Vault libraries still compile as
gateway dependencies; only their standalone Unix-socket binaries and
package-level tests are excluded as top-level targets.
## Common Errors
.agents/skills/helm-dev-environment/SKILL.md
+42 -39
View File
@@ -69,28 +69,15 @@ mise run helm:skaffold:dev
mise run helm:skaffold:run
```
**Supervisor sidecar topology** (build once and leave running):
```bash
mise run helm:skaffold:run:sidecar
```
**Supervisor sidecar topology with TLS/mTLS enabled** (build once and leave running):
```bash
mise run helm:skaffold:run:sidecar-mtls
```
Both commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm
chart. The sidecar profile renders an `openshell-network-init` init container for
nftables setup and an `openshell-supervisor-network` runtime sidecar for proxying.
Binary-aware policy mode runs that sidecar as UID 0 with `SYS_PTRACE` and
`DAC_READ_SEARCH`; relaxed mode can run it as the configured proxy UID, which
must be at least `1000` and distinct from the workload UID. The
sidecar-mTLS profile reuses `ci/values-sidecar.yaml` and restores
`server.disableTls=false` inline for Skaffold. The `pkiInitJob` hook (a pre-install
Job that runs `openshell-gateway generate-certs`) generates mTLS secrets on first
install. The default Skaffold values export gateway and Kubernetes-driver traces to
the collector service installed by `helm:k3s:create`. Envoy Gateway opt-in; see the
Optional Add-ons section below.
The Skaffold flow builds distinct `gateway`, `sandbox`, and `supervisor` images
and deploys the OpenShell Helm chart. The Kubernetes driver creates a
capability-free workload Pod and a directly managed capability-free supervisor
Pod. One namespace-wide NetworkPolicy denies direct egress from every OpenShell
workload Pod. The
`pkiInitJob` hook (a pre-install Job that runs `openshell-gateway generate-certs`)
generates mTLS secrets on first install. The default Skaffold values export
gateway and Kubernetes-driver traces to the collector service installed by
`helm:k3s:create`. Envoy Gateway is opt-in; see the Optional Add-ons section.
The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or
the unified local forwarding task:
@@ -102,9 +89,9 @@ mise run helm:k3s:forward
The task forwards OTLP/gRPC to `http://127.0.0.1:4317` and the trace UI to
`http://127.0.0.1:18888`. When Skaffold has deployed a Kubernetes gateway, it
also forwards the gateway to `http://127.0.0.1:8090`; otherwise it continues
with the collector ports only. A successful plaintext `helm:skaffold:run` or
`helm:skaffold:run:sidecar` registers the gateway under the worktree-specific
k3d cluster name and selects it as the active gateway. Keep the forwarding
with the collector ports only. A successful plaintext `helm:skaffold:run`
registers the gateway under the worktree-specific k3d
cluster name and selects it as the active gateway. Keep the forwarding
task running while using those endpoints.
### Viewing local traces
@@ -134,8 +121,7 @@ create the Secret named `openshell-ha-pg` with a `uri` key, then run
### TLS behaviour
`ci/values-skaffold.yaml` sets `server.disableTls: true`, so Skaffold-based deploys run
plaintext by default. To test sidecar topology with TLS enabled, use
`mise run helm:skaffold:run:sidecar-mtls`.
plaintext by default. Override `server.disableTls=false` to exercise TLS/mTLS.
| Mode | `server.disableTls` | Gateway scheme |
|------|---------------------|----------------|
@@ -188,12 +174,6 @@ openshell sandbox list --gateway-endpoint https://localhost:8090
mise run helm:skaffold:delete
```
For a sidecar-profile deployment:
```bash
mise run helm:skaffold:delete:sidecar
```
### Delete the cluster entirely
```bash
@@ -256,15 +236,19 @@ Key Helm values:
### Keycloak OIDC
One-time setup — only needed once per cluster lifetime:
Initial setup — rerun it whenever you want to rotate the development CA:
```bash
mise run keycloak:k8s:setup
```
This deploys Keycloak (`quay.io/keycloak/keycloak:24.0`) into the `keycloak` namespace,
imports the openshell realm from `scripts/keycloak-realm.json`, and prints a port-forward
command for acquiring tokens from the CLI.
imports the openshell realm from `scripts/keycloak-realm.json`, generates a short-lived
development TLS certificate, and publishes its trust anchor as the
`openshell-keycloak-ca` ConfigMap in the OpenShell namespace. The command prints a
port-forward command for acquiring tokens from the CLI. Rerunning setup rotates the
development certificate and trust anchor; redeploy the gateway afterward so it reloads
the mounted CA bundle.
Then activate OIDC in the OpenShell Helm chart:
1. Uncomment `#- ci/values-keycloak.yaml` in `skaffold.yaml`
@@ -288,12 +272,31 @@ SPIFFE JWT-SVIDs for dynamic provider token grants:
`openshell.local` and adds a `ClusterSPIFFEID` that maps sandbox pod
annotations to `spiffe://openshell.local/openshell/sandbox/<sandbox-id>`.
OpenShell mounts the SPIFFE CSI Workload API socket at
`/spiffe-workload-api/spire-agent.sock` into sandbox pods for provider token
`/spiffe-workload-api/spire-agent.sock` only into supervisor Pods for provider token
grants. Supervisor-to-gateway authentication remains on the Kubernetes
ServiceAccount bootstrap and gateway-minted sandbox JWT path; the selected
Kubernetes compute driver validates the projected token before the gateway
mints its JWT.
### Vault Credential Driver
The `credential-driver-vault` Skaffold profile applies
`ci/values-credential-driver-vault.yaml`. Its external OpenBao/Vault backend
must expose HTTPS at the configured service DNS name and publish the issuing CA
certificate as the `ca.crt` key in the `openbao-ca` ConfigMap. Local e2e uses
OpenBao dev TLS and an `openbao-0` DNS alias matching its generated certificate.
The Helm value
`server.credentialDrivers.vault.caConfigMapName` mounts that key into the
gateway and renders the driver's `ca_bundle` setting. Non-loopback HTTP
addresses fail gateway startup, and hostname verification requires the service
DNS name in the server certificate SANs.
```bash
cd deploy/helm/openshell
skaffold run -p credential-driver-vault
kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200
```
---
## Cluster Lifecycle (stop/start)
@@ -349,10 +352,10 @@ for dependencies still declared in `Chart.yaml`.
| `deploy/helm/openshell/ci/values-gateway.yaml` | Envoy Gateway GRPCRoute + Gateway overlay |
| `deploy/helm/openshell/ci/values-high-availability.yaml` | HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) |
| `deploy/helm/openshell/ci/values-keycloak.yaml` | Keycloak OIDC overlay |
| `deploy/helm/openshell/ci/values-sidecar.yaml` | Supervisor sidecar topology overlay for Kubernetes e2e/dev |
| `deploy/helm/openshell/ci/values-spire.yaml` | SPIFFE/SPIRE provider token grant overlay |
| `deploy/helm/openshell/ci/values-spire-stack.yaml` | SPIRE hardened chart values for local dev |
| `deploy/helm/openshell/ci/values-tls-disabled.yaml` | Lint-only: TLS + auth disabled (reverse-proxy edge termination) |
| `deploy/helm/openshell/ci/values-credential-driver-vault.yaml` | Vault credential-driver validation overlay with HTTPS and private-CA trust |
| `deploy/kube/manifests/envoy-gateway-openshell.yaml` | GatewayClass for Envoy Gateway (`mise run helm:gateway:apply`) |
| `tasks/scripts/helm-k3s-local.sh` | k3d cluster create/delete/start/stop/status |
| `tasks/scripts/keycloak-k8s-setup.sh` | Keycloak deploy + realm import |
| `tasks/scripts/keycloak-k8s-setup.sh` | Keycloak deploy, realm import, and development TLS trust anchor |
.agents/skills/launch-openshell-gator/SKILL.md
+12 -18
View File
@@ -13,7 +13,7 @@ For gator's PR/issue validation policy, load `gator-gate` inside the launched sa
## Non-Negotiable Rules
- Keep normal gator launches supervised: use `--watch --background` and let the in-sandbox supervisor own sleeping and relaunching bounded cycles.
- Keep normal gator launches supervised: use `--watch` and let the in-sandbox supervisor own sleeping and relaunching bounded cycles.
- Do not add passive `sleep` loops in the operator session to watch gator. Check logs or status once, then report the current state or launch a proper watcher outside the model session only when explicitly asked.
- Do not change the default gator model in `scripts/agents/gator/agent.yaml` for experiments. Use `CODEX_MODEL=...` and, if needed, a temporary `--from` Docker context or `--codex-bin` override.
- Do not push to contributor branches, approve, merge, post `/ok to test`, or broaden gator scope unless the operator explicitly authorized that action.
@@ -34,7 +34,6 @@ For gator's PR/issue validation policy, load `gator-gate` inside the launched sa
| `scripts/agents/gator/bin/validate-review-findings` | Enforces the blocker evidence schema and downgrades unsupported hypotheses. |
| `scripts/agents/gator/prompts/gator.md` | Rendered top-level prompt template baked into the payload. |
| `scripts/agents/gator/skills/gator-gate/SKILL.md` | In-sandbox gator state-machine skill. |
| `scripts/agents/gator/logs/` | Background launch and supervisor logs. |
## Preflight
@@ -156,11 +155,12 @@ sandbox_name="gator-pr-${pr_number}-supervised"
--gateway "$gateway_name" \
--name "$sandbox_name" \
--watch \
--background \
"Review and monitor PR #${pr_number} through the gator-gate workflow. Scope this invocation only to PR #${pr_number}."
```
The launcher queries gateway's selected compute driver, builds gator image in matching Docker or Podman image store, stages immutable payload, imports provider profiles, configures provider credentials and refresh, creates and uploads sandbox payload, then starts agent supervisor with `sandbox exec`. It writes a background log under `scripts/agents/gator/logs/`. `CONTAINER_ENGINE`, when set, must match gateway driver.
The launcher queries the gateway's selected compute driver, builds the gator image in the matching Docker or Podman image store, stages the immutable payload, imports provider profiles, configures provider credentials and refresh, and starts the agent supervisor as the sandbox's canonical main process. The detached main process survives loss of the host CLI connection and reconnects to a restarted gateway. Unless `--keep` is set, the sandbox is marked ephemeral so the gateway deletes it after the supervisor exits. `CONTAINER_ENGINE`, when set, must match the gateway driver.
The launcher streams image-build and provisioning output until the detached workload is ready, then exits. Use `openshell logs <sandbox-name>` or the TUI for runtime output.
### Launch An Issue Or Issue/PR Pair
@@ -177,7 +177,6 @@ sandbox_name="gator-issue-${issue_number}-supervised"
--gateway "$gateway_name" \
--name "$sandbox_name" \
--watch \
--background \
"Run gator on issue #${issue_number}. Scope this invocation only to issue #${issue_number}."
```
@@ -198,7 +197,6 @@ sandbox_name="gator-pr-${pr_number}-supervised"
--gateway "$gateway_name" \
--name "$sandbox_name" \
--watch \
--background \
"Review and monitor PR #${pr_number} with linked issue #${issue_number} through the gator-gate workflow. Scope this invocation only to PR #${pr_number} and issue #${issue_number}."
```
@@ -219,7 +217,6 @@ sandbox_name="gator-pr-${pr_number}-supervised"
--gateway "$gateway_name" \
--name "$sandbox_name" \
--watch \
--background \
"Review and monitor PR #${pr_number} through the gator-gate workflow. Scope this invocation only to PR #${pr_number}. The operator explicitly authorizes applying the test:e2e label, posting /ok to test for the current head SHA, and rerunning the relevant current-head workflow when the E2E Label Help bot says that is required."
```
@@ -241,7 +238,6 @@ CODEX_MODEL=gpt-5.6-sol \
--gateway "$gateway_name" \
--name "$sandbox_name" \
--watch \
--background \
"Review and monitor PR #${pr_number} through the gator-gate workflow. Scope this invocation only to PR #${pr_number}. This launch is intentionally testing Codex model gpt-5.6-sol via the CLI launcher."
```
@@ -266,7 +262,6 @@ CODEX_MODEL=gpt-5.6-sol \
--name "$sandbox_name" \
--from "$tmp_context" \
--watch \
--background \
"Review and monitor PR #${pr_number} through the gator-gate workflow. Scope this invocation only to PR #${pr_number}."
```
@@ -274,19 +269,15 @@ CODEX_MODEL=gpt-5.6-sol \
### Read The Launch Result
The launcher prints the log path when `--background` is used:
```text
Started in background. Log: scripts/agents/gator/logs/<sandbox-name>.log
```
Read that file directly. Important markers:
The launcher streams image-build and provisioning output to the terminal. Important markers:
- `Built image ...` means the local image build completed.
- `Created sandbox: <name>` means OpenShell accepted the sandbox.
- `openshell-agent: starting watch cycle` means the in-sandbox supervisor began a bounded cycle.
- `OpenAI Codex v...` plus `model: ...` confirms the Codex CLI and model actually used.
- `OPENSHELL_AGENT_RESULT {...}` is the bounded-cycle sentinel. In watch mode, the supervisor sleeps and relaunches after this line.
- `/sandbox/.openshell-agent/status.json` is the atomic current state snapshot. Its `result.notes` field is Gator's plain-language diagnosis and next action for that cycle.
- `/sandbox/.openshell-agent/history.jsonl` contains the latest 100 supervisor transitions, including active-cycle starts and completed cycle results.
- `openshell-agent: still running watch cycle ...` is a heartbeat during long active model cycles.
- `review_feedback_lookup_failed` means Gator could not build the required cross-SHA feedback ledger and deliberately skipped a context-free review.
@@ -314,6 +305,11 @@ If `sandbox get` is not supported by the local CLI shape, use `openshell sandbox
| `status=terminal_failure` | Unrecoverable or stale immutable payload. | Inspect the reason; rebuild/relaunch for `stale_gator_payload`. |
| `status=complete` | Target closed, merged, or one-shot complete. | Delete sandbox if no longer needed. |
Prefer the state snapshot over scraping transient `/tmp` cycle output. Use the
history file to tell whether a failure is repeating or whether the supervisor
has begun a fresh cycle. Runtime logs remain useful for full command output and
transport diagnostics.
## Restarting A Gator
Restart when the payload must change, the sandbox is wedged without a sentinel, the model/tooling version changed, or a transient failure repeats past the useful retry point.
@@ -339,7 +335,6 @@ openshell --gateway "$gateway_name" sandbox delete "$sandbox_name"
--gateway "$gateway_name" \
--name "$sandbox_name" \
--watch \
--background \
"<same scoped operator prompt, updated only with the reason for relaunch>"
```
@@ -401,7 +396,6 @@ When you launch or inspect gator, report:
- Sandbox name.
- Gateway name.
- Log path.
- Target issue/PR scope.
- Harness and model when relevant.
- Whether image build and sandbox creation succeeded.
.agents/skills/sbom/SKILL.md
+1 -1
View File
@@ -15,7 +15,7 @@ The OpenShell SBOM tooling produces source-tree CycloneDX JSON SBOMs using Syft,
SBOMs are **release artifacts only** -- they are generated on demand and not committed to the repository. Output lands in `deploy/sbom/output/` (gitignored).
Pushed gateway and supervisor images carry an SPDX SBOM and minimal SLSA provenance as OCI attestations. Branch E2E, Release Dev, and Release Tag image binaries embed cargo-auditable metadata, so their image SBOMs include linked Rust crates.
Pushed gateway, sandbox, and supervisor images carry an SPDX SBOM and minimal SLSA provenance as OCI attestations. Branch E2E, Release Dev, and Release Tag image binaries embed cargo-auditable metadata, so their image SBOMs include linked Rust crates.
## Prerequisites
.agents/skills/test-release-canary/SKILL.md
+12 -9
View File
@@ -13,18 +13,20 @@ The Release Canary (`.github/workflows/release-canary.yml`) smoke-tests the arti
| Job | Runner | Verifies |
|---|---|---|
| `macos` | `macos-latest-xlarge` | `install.sh` resolves the Homebrew formula, brew installs the cask, and `openshell status` reaches the brew-services–backed local gateway with the VM driver. |
| `ubuntu` | `ubuntu-latest` | `install.sh` installs the Debian package, the post-install systemd user service starts, and `openshell status` reaches the local gateway with the Docker driver. |
| `fedora` | `fedora:latest` container | `install.sh` installs the RPM packages, the local gateway starts under Podman, and `openshell status` succeeds. |
| `ubuntu-snap` | `ubuntu-latest` | Downloads the Snap artifact from Release Dev, installs it with `--dangerous`, connects the required interfaces, and waits up to 30 seconds for the recovered local gateway. |
| `kubernetes` | `ubuntu-latest` + kind | `helm install oci://ghcr.io/nvidia/openshell/helm-chart --version 0.0.0-dev` succeeds in a kind cluster, the gateway pod becomes Ready, port-forward exposes 8080, and the released CLI registers the in-cluster gateway and runs `openshell status` against it. |
| `macos` | `macos-latest-xlarge` | Installs the dev Homebrew artifacts, reaches the VM gateway, and creates, executes in, and deletes a sandbox. |
| `ubuntu` | `ubuntu-latest` | Installs the dev Debian package, reaches the Docker gateway, and creates, executes in, and deletes a sandbox. |
| `fedora` | `fedora:latest` container | Installs the dev RPM packages, reaches the Podman gateway, and creates, executes in, and deletes a sandbox. |
| `ubuntu-snap` | `ubuntu-latest` | Installs the Release Dev Snap, connects its interfaces, reaches the Docker gateway, and creates, executes in, and deletes a sandbox. |
| `kubernetes` | `ubuntu-latest` + kind | Installs the dev Helm chart, reaches the in-cluster gateway, and creates, executes in, and deletes a sandbox using the published runtime images. |
All canary jobs disable anonymous OpenShell telemetry. Host package jobs inject
`OPENSHELL_TELEMETRY_ENABLED=false` through the service environment, and the
Kubernetes job installs with `server.telemetryEnabled=false`, so smoke traffic
does not contribute to product usage metrics.
`install.sh` defaults to the *latest tagged* release — the canary is therefore checking that the most recent public release still installs, not the just-published `dev` build. The `kubernetes` job is the exception: it pins to `0.0.0-dev` chart + `:dev` images.
The workflow sets `OPENSHELL_VERSION=dev`, so every `install.sh` job consumes the
rolling dev release produced by the triggering workflow. Kubernetes pins the
matching `0.0.0-dev` chart and `:dev` images.
The host-package jobs exercise fresh installs, not upgrades from a persisted
schema-v1 gateway config. Validate Homebrew and RPM exact-default migration with
@@ -83,7 +85,7 @@ gh run view <run-id> --log-failed
## Iterating on the canary itself
When you change `release-canary.yml` on a branch, a manual dispatch on that branch tests *your branch's workflow logic* against *main's published artifacts* (`0.0.0-dev` chart, `:dev` images, latest tagged install.sh assets). This is what you want for iterating on the canary — you're validating that the canary still works against known-good artifacts.
When you change `release-canary.yml` on a branch, a manual dispatch on that branch tests *your branch's workflow logic* against *main's published dev artifacts* (`0.0.0-dev` chart, `:dev` images, and the `dev` GitHub release). This is what you want for iterating on the canary — you're validating that the canary still works against known-good artifacts.
Note `install.sh` is pulled from `raw.githubusercontent.com/NVIDIA/OpenShell/${head_sha}/install.sh`, so changes to `install.sh` on your branch *are* exercised even though the binaries it downloads are from the latest public tag.
@@ -92,7 +94,7 @@ Note `install.sh` is pulled from `raw.githubusercontent.com/NVIDIA/OpenShell/${h
`Release Dev` publishes two chart versions for every dev build (see `.github/actions/release-helm-oci/action.yml:89-102`):
- `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` — floating, overwritten on every main push.
- `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev.<sha>` — immutable, `appVersion` set to the same SHA so it pulls `ghcr.io/nvidia/openshell/gateway:<sha>` and `:supervisor:<sha>`.
- `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev.<sha>` — immutable, `appVersion` set to the same SHA so it pulls the matching `gateway`, `sandbox`, and `supervisor` images.
To smoke-test the chart for a specific dev build, dispatch `Release Dev` on the branch first, then run the kind canary steps locally pointed at the SHA-pinned chart (see "Local kind reproduction" below). The release-canary workflow itself does not currently expose `chart_version` / `image_tag` inputs.
@@ -134,7 +136,8 @@ Loopback registration auto-derives the gateway name to `openshell` if `--name` i
| Symptom | Likely cause | Where to look |
|---|---|---|
| `macos`/`ubuntu`/`fedora` job fails on `install.sh` | Latest tagged release missing an asset, checksum mismatch, or `install.sh` regression on this branch. | Job log around the `curl … install.sh \| sh` step. |
| `macos`/`ubuntu`/`fedora` job fails on `install.sh` | Dev release missing an asset, checksum mismatch, or `install.sh` regression on this branch. | Job log around the `curl … install.sh \| sh` step. |
| Sandbox create or exec fails | Published sandbox and supervisor artifacts are missing, incompatible, or cannot establish the protected runtime channel. | Gateway logs plus Docker, Podman, VM, Snap, or Kubernetes runtime diagnostics for the job. |
| `macos`/`ubuntu`/`fedora` job fails on `openshell status` | Local gateway service did not start (systemd/brew/podman). Often a driver issue. | Service logs in the job log; `OPENSHELL_COMPUTE_DRIVER` env in the "Ensure …" step. |
| `ubuntu-snap` fails after interface connection | The gateway did not recover after Docker became available, or did not become reachable within the 30-second bound. | Failure diagnostics dump Snap service/connection/change state, gateway and snapd journals, Snap logs, and port 17670 listeners. |
| `kubernetes` job fails on `helm install --wait` | Chart did not deploy in 5 min — usually image pull failure or readiness probe failing. | "Diagnostics on failure" step dumps `helm status`, manifest, pod describe, pod logs. |
.github/actions/build-rust-binary/action.yml
+19 -2
View File
@@ -29,6 +29,10 @@ inputs:
description: Additional flags passed to cargo build
required: false
default: ""
interpreter:
description: Expected ELF program interpreter, "none" for static, or "skip"
required: false
default: skip
runs:
using: composite
@@ -71,11 +75,24 @@ runs:
env:
INPUTS_BINARY: ${{ inputs.binary }}
INPUTS_TRIPLE: ${{ inputs.triple }}
INPUTS_INTERPRETER: ${{ inputs.interpreter }}
run: |
binary="target/${INPUTS_TRIPLE}/release/${INPUTS_BINARY}"
# Confirm the binary runs and reports the expected name.
target/${INPUTS_TRIPLE}/release/${INPUTS_BINARY} --version | grep -q "^${INPUTS_BINARY} "
"$binary" --version | grep -q "^${INPUTS_BINARY} "
# Confirm Syft can decode the embedded cargo-auditable metadata.
SYFT_CHECK_FOR_APP_UPDATE=false syft file:target/${INPUTS_TRIPLE}/release/${INPUTS_BINARY} -o cyclonedx-json | grep 'pkg:cargo/' > /dev/null
SYFT_CHECK_FOR_APP_UPDATE=false syft "file:${binary}" -o cyclonedx-json | grep 'pkg:cargo/' > /dev/null
if [ "$INPUTS_INTERPRETER" != skip ]; then
actual_interpreter="$(readelf -l "$binary" | sed -n 's/.*Requesting program interpreter: \(.*\)]/\1/p')"
expected_interpreter="$INPUTS_INTERPRETER"
if [ "$expected_interpreter" = none ]; then
expected_interpreter=""
fi
if [ "$actual_interpreter" != "$expected_interpreter" ]; then
echo "error: expected interpreter '${expected_interpreter:-<none>}', got '${actual_interpreter:-<none>}'" >&2
exit 1
fi
fi
- name: Upload ${{ inputs.binary }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
.github/actions/check-job-results/action.yml
-1
View File
@@ -8,7 +8,6 @@ inputs:
results:
description: JSON-encoded GitHub Actions needs context
required: true
runs:
using: composite
steps:
.github/actions/setup-e2e-kind/action.yml
+2 -2
View File
@@ -14,7 +14,7 @@ inputs:
images:
description: Space-separated OpenShell image components to load
required: false
default: gateway supervisor
default: gateway sandbox supervisor
registry:
description: Container registry and namespace
required: false
@@ -71,7 +71,7 @@ runs:
set -euo pipefail
for component in $IMAGE_COMPONENTS; do
case "$component" in
gateway | supervisor) ;;
gateway | sandbox | supervisor) ;;
*) echo "ERROR: unsupported OpenShell image component: $component" >&2; exit 1 ;;
esac
image="${REGISTRY}/${component}:${IMAGE_TAG}"
.github/workflows/branch-checks.yml
+5 -5
View File
@@ -179,8 +179,8 @@ jobs:
tasks/scripts/verify-telemetry-compiled-out.sh present target/debug/openshell-gateway
cargo build -p openshell-gateway --bin openshell-gateway --no-default-features --features defaults-without-telemetry
tasks/scripts/verify-telemetry-compiled-out.sh absent target/debug/openshell-gateway
cargo build -p openshell-sandbox --bin openshell-sandbox --no-default-features --features defaults-without-telemetry
tasks/scripts/verify-telemetry-compiled-out.sh absent target/debug/openshell-sandbox
cargo build -p openshell-supervisor --bin openshell-supervisor --no-default-features --features defaults-without-telemetry
tasks/scripts/verify-telemetry-compiled-out.sh absent target/debug/openshell-supervisor
- name: Verify selective gateway compute-driver builds
run: |
@@ -197,12 +197,12 @@ jobs:
- name: Verify system CA roots build mode compiles and excludes bundled Mozilla roots
run: |
cargo check -p openshell-sandbox --all-targets --no-default-features --features system-ca-roots
if cargo tree -p openshell-sandbox -i webpki-roots --no-default-features --features system-ca-roots 2>/dev/null | grep -q webpki-roots; then
cargo check -p openshell-supervisor --all-targets --no-default-features --features system-ca-roots
if cargo tree -p openshell-supervisor -i webpki-roots --no-default-features --features system-ca-roots 2>/dev/null | grep -q webpki-roots; then
echo "ERROR: webpki-roots found in system CA roots build" >&2
exit 1
fi
if cargo tree -p openshell-sandbox -i webpki-root-certs --no-default-features --features system-ca-roots 2>/dev/null | grep -q webpki-root-certs; then
if cargo tree -p openshell-supervisor -i webpki-root-certs --no-default-features --features system-ca-roots 2>/dev/null | grep -q webpki-root-certs; then
echo "ERROR: webpki-root-certs found in system CA roots build" >&2
exit 1
fi
.github/workflows/branch-e2e.yml
+33 -28
View File
@@ -68,6 +68,7 @@ jobs:
else
run_any_e2e=false
fi
{
echo "run_core_e2e=$run_core_e2e"
echo "run_gpu_e2e=$run_gpu_e2e"
@@ -127,11 +128,11 @@ jobs:
cargo-version: ${{ needs.version.outputs.cargo }}
secrets: inherit
build-sandbox:
build-runtimes:
needs: version
permissions:
contents: read
uses: ./.github/workflows/build-sandbox-binaries.yml
uses: ./.github/workflows/build-runtime-binaries.yml
with:
cargo-version: ${{ needs.version.outputs.cargo }}
secrets: inherit
@@ -211,7 +212,7 @@ jobs:
secrets: inherit
build-vm-driver:
needs: [pr_metadata, version, build-sandbox]
needs: [pr_metadata, version, build-runtimes]
if: needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
contents: read
@@ -237,8 +238,21 @@ jobs:
target-suffix: unknown-linux-gnu
secrets: inherit
build-sandbox-image:
needs: [pr_metadata, build-runtimes]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_any_e2e == 'true'
permissions:
contents: read
packages: write
uses: ./.github/workflows/docker-build.yml
with:
component: sandbox
binary: openshell-sandbox
target-suffix: unknown-linux-musl
secrets: inherit
build-supervisor-image:
needs: [pr_metadata, build-sandbox]
needs: [pr_metadata, build-runtimes]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_any_e2e == 'true'
permissions:
contents: read
@@ -246,12 +260,12 @@ jobs:
uses: ./.github/workflows/docker-build.yml
with:
component: supervisor
binary: openshell-sandbox
target-suffix: unknown-linux-musl
binary: openshell-supervisor
target-suffix: unknown-linux-gnu
secrets: inherit
docker-e2e:
needs: [pr_metadata, build-cli, build-conformance, build-gateway, build-supervisor-image]
needs: [pr_metadata, build-cli, build-conformance, build-gateway, build-sandbox-image, build-supervisor-image]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
@@ -264,7 +278,7 @@ jobs:
conformance-artifact-prefix: openshell-conformance
podman-e2e:
needs: [pr_metadata, build-cli, build-conformance, build-gateway, build-supervisor-image]
needs: [pr_metadata, build-cli, build-conformance, build-gateway, build-sandbox-image, build-supervisor-image]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
@@ -287,7 +301,7 @@ jobs:
conformance-artifact-prefix: openshell-conformance
docker-external-driver-e2e:
needs: [pr_metadata, build-cli, build-conformance, build-gateway-plain, build-driver-docker, build-supervisor-image]
needs: [pr_metadata, build-cli, build-conformance, build-gateway-plain, build-driver-docker, build-sandbox-image, build-supervisor-image]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
@@ -304,7 +318,7 @@ jobs:
[{"suite":"external-driver","cmd":"mise run --no-deps --skip-deps e2e:docker:external-driver","apt_packages":"openssh-client","python_proto":false,"mcp":false}]
podman-external-driver-e2e:
needs: [pr_metadata, build-cli, build-conformance, build-gateway-plain, build-driver-podman, build-supervisor-image]
needs: [pr_metadata, build-cli, build-conformance, build-gateway-plain, build-driver-podman, build-sandbox-image, build-supervisor-image]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
@@ -334,7 +348,7 @@ jobs:
conformance-artifact-prefix: openshell-conformance
gpu-e2e:
needs: [pr_metadata, build-cli, build-conformance, build-gateway, build-supervisor-image]
needs: [pr_metadata, build-cli, build-conformance, build-gateway, build-sandbox-image, build-supervisor-image]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_gpu_e2e == 'true'
permissions:
actions: read
@@ -346,7 +360,7 @@ jobs:
conformance-artifact-prefix: openshell-conformance
kubernetes-e2e:
needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-supervisor-image]
needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-sandbox-image, build-supervisor-image]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
strategy:
fail-fast: false
@@ -354,16 +368,8 @@ jobs:
include:
- agent_sandbox_api: v1beta1
agent_sandbox_version: v0.5.0
topology: combined
extra_helm_values: ""
- agent_sandbox_api: v1alpha1
agent_sandbox_version: v0.4.6
topology: combined
extra_helm_values: ""
- agent_sandbox_api: v1beta1
agent_sandbox_version: v0.5.0
topology: sidecar
extra_helm_values: deploy/helm/openshell/ci/values-sidecar.yaml
permissions:
actions: read
contents: read
@@ -371,13 +377,12 @@ jobs:
uses: ./.github/workflows/e2e-kubernetes-test.yml
with:
image-tag: ${{ github.sha }}
job-name: Kubernetes E2E (Rust smoke, ${{ matrix.topology }}, Agent Sandbox ${{ matrix.agent_sandbox_api }})
job-name: Kubernetes E2E (Rust smoke, Agent Sandbox ${{ matrix.agent_sandbox_api }})
agent-sandbox-version: ${{ matrix.agent_sandbox_version }}
extra-helm-values: ${{ matrix.extra_helm_values }}
conformance-artifact-prefix: openshell-conformance
kubernetes-workspace-managed-e2e:
needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-supervisor-image]
needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-sandbox-image, build-supervisor-image]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
@@ -391,7 +396,7 @@ jobs:
conformance-artifact-prefix: openshell-conformance
kubernetes-external-driver-e2e:
needs: [pr_metadata, build-cli, build-conformance, build-gateway-plain, build-driver-kubernetes, build-supervisor-image]
needs: [pr_metadata, build-cli, build-conformance, build-gateway-plain, build-driver-kubernetes, build-sandbox-image, build-supervisor-image]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
@@ -404,11 +409,11 @@ jobs:
e2e-task: e2e:kubernetes:external-driver
gateway-artifact: openshell-gateway-plain-x86_64-unknown-linux-gnu
external-driver-binary: openshell-driver-kubernetes
cluster-images: supervisor
cluster-images: sandbox supervisor
conformance-artifact-prefix: openshell-conformance
kubernetes-workspace-operator-e2e:
needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-supervisor-image]
needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-sandbox-image, build-supervisor-image]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
@@ -422,7 +427,7 @@ jobs:
conformance-artifact-prefix: openshell-conformance
kubernetes-ha-e2e:
needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-supervisor-image]
needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-sandbox-image, build-supervisor-image]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_ha_e2e == 'true'
permissions:
actions: read
@@ -437,7 +442,7 @@ jobs:
conformance-artifact-prefix: openshell-conformance
kubernetes-credential-drivers-e2e:
needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-supervisor-image]
needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-sandbox-image, build-supervisor-image]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_credential_drivers_e2e == 'true'
permissions:
actions: read
.github/workflows/build-binaries.yml
+6
View File
@@ -33,6 +33,11 @@ on:
required: false
type: string
default: ""
interpreter:
description: Expected ELF program interpreter, "none" for static, or "skip"
required: false
type: string
default: skip
checkout-ref:
required: false
type: string
@@ -67,3 +72,4 @@ jobs:
image-tag: ${{ inputs['image-tag'] }}
artifact-name: ${{ inputs['artifact-name'] }}
extra-cargo-flags: ${{ inputs['extra-cargo-flags'] }}
interpreter: ${{ inputs.interpreter }}
.github/workflows/build-sandbox-binaries.yml → .github/workflows/build-runtime-binaries.yml
+20 -3
View File
@@ -1,7 +1,7 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
name: Build Sandbox Binaries
name: Build Runtime Binaries
on:
workflow_call:
@@ -31,15 +31,32 @@ jobs:
include:
- triple: x86_64-unknown-linux-musl
runner: linux-amd64-cpu8
package: openshell-sandbox
binary: openshell-sandbox
interpreter: none
- triple: x86_64-unknown-linux-gnu
runner: linux-amd64-cpu8
package: openshell-supervisor
binary: openshell-supervisor
interpreter: /lib64/ld-linux-x86-64.so.2
- triple: aarch64-unknown-linux-musl
runner: linux-arm64-cpu8
package: openshell-sandbox
binary: openshell-sandbox
interpreter: none
- triple: aarch64-unknown-linux-gnu
runner: linux-arm64-cpu8
package: openshell-supervisor
binary: openshell-supervisor
interpreter: /lib/ld-linux-aarch64.so.1
uses: ./.github/workflows/build-binaries.yml
with:
package: openshell-sandbox
binary: openshell-sandbox
package: ${{ matrix.package }}
binary: ${{ matrix.binary }}
triple: ${{ matrix.triple }}
runner: ${{ matrix.runner }}
cargo-version: ${{ inputs.cargo-version }}
image-tag: ${{ inputs.image-tag }}
interpreter: ${{ matrix.interpreter }}
checkout-ref: ${{ inputs.checkout-ref }}
secrets: inherit
.github/workflows/build-vm-driver.yml
+66
View File
@@ -25,8 +25,48 @@ permissions:
contents: read
jobs:
host-supervisor-macos:
name: native host supervisor (aarch64-apple-darwin)
permissions:
contents: read
uses: ./.github/workflows/build-binaries.yml
with:
package: openshell-supervisor
binary: openshell-supervisor
triple: aarch64-apple-darwin
runner: macos-15-xlarge
cargo-version: ${{ inputs.cargo-version }}
image-tag: ${{ inputs.image-tag }}
checkout-ref: ${{ inputs.checkout-ref }}
secrets: inherit
guest-init:
name: guest init (${{ matrix.triple }})
strategy:
matrix:
include:
- arch: x86_64
triple: x86_64-unknown-linux-musl
runner: linux-amd64-cpu8
- arch: aarch64
triple: aarch64-unknown-linux-musl
runner: linux-arm64-cpu8
uses: ./.github/workflows/build-binaries.yml
with:
package: openshell-driver-vm
binary: openshell-vm-init
triple: ${{ matrix.triple }}
runner: ${{ matrix.runner }}
cargo-version: ${{ inputs.cargo-version }}
image-tag: ${{ inputs.image-tag }}
extra-cargo-flags: --no-default-features
interpreter: none
checkout-ref: ${{ inputs.checkout-ref }}
secrets: inherit
build:
name: openshell-driver-vm (${{ matrix.triple }})
needs: [guest-init, host-supervisor-macos]
strategy:
matrix:
include:
@@ -59,15 +99,41 @@ jobs:
name: openshell-sandbox-${{ matrix.arch }}-unknown-linux-musl
path: sandbox
- name: Download openshell-supervisor
if: endsWith(matrix.triple, '-unknown-linux-gnu')
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: openshell-supervisor-${{ matrix.arch }}-unknown-linux-gnu
path: supervisor
- name: Download native macOS host supervisor
if: endsWith(matrix.triple, '-apple-darwin')
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: openshell-supervisor-aarch64-apple-darwin
path: host-supervisor
- name: Download VM guest init
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: openshell-vm-init-${{ matrix.arch }}-unknown-linux-musl
path: vm-init
- name: Build VM runtime
run: nix build .#vm-runtime
- name: Assemble compressed VM runtime
run: |
compressed_dir="${RUNNER_TEMP}/vm-runtime-compressed"
host_supervisor="supervisor/openshell-supervisor"
if [[ "${{ matrix.triple }}" == *-apple-darwin ]]; then
host_supervisor="host-supervisor/openshell-supervisor"
fi
install -d "$compressed_dir"
cp result/compressed/*.zst "$compressed_dir/"
zstd -19 -T1 sandbox/openshell-sandbox -o "$compressed_dir/openshell-sandbox.zst"
zstd -19 -T1 "$host_supervisor" -o "$compressed_dir/openshell-supervisor.zst"
zstd -19 -T1 vm-init/openshell-vm-init -o "$compressed_dir/openshell-vm-init.zst"
- name: Build openshell-driver-vm
uses: ./.github/actions/build-rust-binary
.github/workflows/e2e-docker-test.yml
+1
View File
@@ -62,6 +62,7 @@ jobs:
MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
IMAGE_TAG: ${{ inputs.image-tag }}
OPENSHELL_REGISTRY: ghcr.io/nvidia/openshell
OPENSHELL_SANDBOX_RUNTIME_IMAGE: ${{ format('ghcr.io/nvidia/openshell/sandbox:{0}', inputs.image-tag) }}
OPENSHELL_SUPERVISOR_IMAGE: ${{ format('ghcr.io/nvidia/openshell/supervisor:{0}', inputs.image-tag) }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
.github/workflows/e2e-kubernetes-test.yml
+1 -1
View File
@@ -76,7 +76,7 @@ on:
description: "Space-separated published images to preload into kind"
required: false
type: string
default: "gateway supervisor"
default: "gateway sandbox supervisor"
permissions:
actions: read
.github/workflows/e2e-label-help.yml
+2 -2
View File
@@ -42,7 +42,7 @@ jobs:
case "$LABEL_NAME" in
test:e2e)
suite_summary="the standard E2E suite"
build_summary="gateway and supervisor images"
build_summary="gateway, sandbox, and supervisor images"
status_summary="The matching required CI gate status on this PR will flip green automatically once the run finishes."
;;
test:e2e-gpu)
@@ -52,7 +52,7 @@ jobs:
;;
test:e2e-kubernetes)
suite_summary="Kubernetes HA and credential-driver E2E"
build_summary="gateway and supervisor images"
build_summary="gateway, sandbox, and supervisor images"
status_summary="This is an optional proof-of-life suite; failures are visible in the workflow run but do not publish a required CI gate status."
;;
*) echo "Unrecognized label $LABEL_NAME"; exit 1 ;;
.github/workflows/e2e-podman-test.yml
+1
View File
@@ -49,6 +49,7 @@ jobs:
IMAGE_TAG: ${{ inputs.image-tag }}
MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OPENSHELL_REGISTRY: ghcr.io/nvidia/openshell
OPENSHELL_SANDBOX_RUNTIME_IMAGE: ${{ format('ghcr.io/nvidia/openshell/sandbox:{0}', inputs.image-tag) }}
OPENSHELL_SUPERVISOR_IMAGE: ${{ format('ghcr.io/nvidia/openshell/supervisor:{0}', inputs.image-tag) }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
.github/workflows/package-release-binaries.yml
+7 -1
View File
@@ -39,9 +39,15 @@ jobs:
package: gateway-binary-macos
- artifact: openshell-sandbox-x86_64-unknown-linux-musl
binary: openshell-sandbox
package: supervisor-binary-linux-amd64
package: sandbox-binary-linux-amd64
- artifact: openshell-sandbox-aarch64-unknown-linux-musl
binary: openshell-sandbox
package: sandbox-binary-linux-arm64
- artifact: openshell-supervisor-x86_64-unknown-linux-gnu
binary: openshell-supervisor
package: supervisor-binary-linux-amd64
- artifact: openshell-supervisor-aarch64-unknown-linux-gnu
binary: openshell-supervisor
package: supervisor-binary-linux-arm64
- artifact: openshell-driver-vm-x86_64-unknown-linux-gnu
binary: openshell-driver-vm
.github/workflows/release-auto-tag.yml
+37 -13
View File
@@ -5,8 +5,9 @@ name: Release Auto-Tag
on:
workflow_dispatch: {}
# schedule:
# - cron: "0 14 * * 1-5" # 7 AM PDT, weekdays only
schedule:
- cron: "0 14 * * 1-5"
timezone: "Europe/Zurich"
permissions:
contents: write
@@ -19,17 +20,40 @@ concurrency:
jobs:
create-tag:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: main
fetch-depth: 0
- name: Determine next patch version
- name: Determine next pre-release version
id: version
run: |
latest=$(git tag -l 'v*.*.*' --sort=-v:refname | head -1)
if [ -z "$latest" ]; then
echo "::error::No existing v*.*.* tags found"
set -euo pipefail
tags=$(git tag -l)
latest=$(printf '%s\n' "$tags" | \
{ grep -E '^v[0-9]+\.[0-9]+\.[0-9]+(-pre\.[1-9][0-9]*)?$' || true; } | \
sort -V | tail -1)
if [[ ! "$latest" =~ ^(v[0-9]+\.[0-9]+\.[0-9]+)-pre\.([1-9][0-9]*)$ ]]; then
echo "No active pre-release series — skipping tag creation"
echo "skip=true" >> "$GITHUB_OUTPUT"
exit 0
fi
base=${BASH_REMATCH[1]}
number=${BASH_REMATCH[2]}
if ! git show-ref --verify --quiet "refs/tags/${base}-pre.1"; then
echo "No ${base}-pre.1 seed — skipping tag creation"
echo "skip=true" >> "$GITHUB_OUTPUT"
exit 0
fi
if git show-ref --verify --quiet "refs/tags/${base}"; then
echo "$base is already stable — skipping tag creation"
echo "skip=true" >> "$GITHUB_OUTPUT"
exit 0
fi
if ! git merge-base --is-ancestor "$latest" HEAD; then
echo "::error::Latest pre-release $latest is not an ancestor of main"
exit 1
fi
echo "Latest tag: $latest"
@@ -43,10 +67,7 @@ jobs:
exit 0
fi
major=$(echo "$latest" | sed 's/^v//' | cut -d. -f1)
minor=$(echo "$latest" | sed 's/^v//' | cut -d. -f2)
patch=$(echo "$latest" | sed 's/^v//' | cut -d. -f3)
next="v${major}.${minor}.$((patch + 1))"
next="${base}-pre.$((number + 1))"
if git tag -l "$next" | grep -q .; then
echo "::error::Tag $next already exists"
@@ -58,15 +79,18 @@ jobs:
- name: Create and push tag
if: steps.version.outputs.skip != 'true'
env:
NEXT_TAG: ${{ steps.version.outputs.next }}
run: |
git tag ${{ steps.version.outputs.next }}
git push origin ${{ steps.version.outputs.next }}
git tag "$NEXT_TAG"
git push origin "refs/tags/$NEXT_TAG"
- name: Trigger Release Tag workflow
if: steps.version.outputs.skip != 'true'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
NEXT_TAG: ${{ steps.version.outputs.next }}
run: |
gh workflow run release-tag.yml \
--ref main \
-f tag=${{ steps.version.outputs.next }}
-f tag="$NEXT_TAG"
.github/workflows/release-canary.yml
+28
View File
@@ -20,6 +20,7 @@ defaults:
shell: bash
env:
OPENSHELL_VERSION: dev
OPENSHELL_TELEMETRY_ENABLED: "false"
jobs:
@@ -36,8 +37,13 @@ jobs:
- name: Install and check status
run: |
set -euo pipefail
curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh | sh
openshell status
sandbox="release-canary-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
openshell sandbox create --name "$sandbox" --detach
openshell sandbox exec --name "$sandbox" --no-tty -- true
openshell sandbox delete "$sandbox"
ubuntu:
name: Ubuntu Docker
@@ -59,8 +65,13 @@ jobs:
- name: Install and check status
run: |
set -euo pipefail
curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh | sh
openshell status
sandbox="release-canary-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
openshell sandbox create --name "$sandbox" --detach
openshell sandbox exec --name "$sandbox" --no-tty -- true
openshell sandbox delete "$sandbox"
fedora:
name: Fedora RPM
@@ -140,7 +151,9 @@ jobs:
HOME=/root \
XDG_RUNTIME_DIR=/run/user/0 \
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus \
OPENSHELL_VERSION="$OPENSHELL_VERSION" \
OPENSHELL_TELEMETRY_ENABLED="$OPENSHELL_TELEMETRY_ENABLED" \
CANARY_SANDBOX="release-canary-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" \
INSTALL_SH_URL="https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh" \
bash -s <<'EOF'
set -euo pipefail
@@ -150,6 +163,9 @@ jobs:
podman info
curl -LsSf "${INSTALL_SH_URL}" | sh
openshell status
openshell sandbox create --name "$CANARY_SANDBOX" --detach
openshell sandbox exec --name "$CANARY_SANDBOX" --no-tty -- true
openshell sandbox delete "$CANARY_SANDBOX"
EOF
- name: Stop Fedora systemd container
@@ -216,6 +232,14 @@ jobs:
echo "Gateway did not become ready within 30 seconds" >&2
exit 1
- name: Create and exercise a sandbox
run: |
set -euo pipefail
sandbox="release-canary-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
openshell sandbox create --name "$sandbox" --detach
openshell sandbox exec --name "$sandbox" --no-tty -- true
openshell sandbox delete "$sandbox"
- name: Collect Snap diagnostics
if: failure()
run: |
@@ -309,6 +333,10 @@ jobs:
set -euo pipefail
openshell gateway add http://127.0.0.1:8080 --local --name "$KIND_GATEWAY_NAME"
openshell status
sandbox="release-canary-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
openshell sandbox create --name "$sandbox" --detach
openshell sandbox exec --name "$sandbox" --no-tty -- true
openshell sandbox delete "$sandbox"
- name: Diagnostics on failure
if: failure()
.github/workflows/release-dev.yml
+79 -15
View File
@@ -99,18 +99,18 @@ jobs:
image-tag: dev
secrets: inherit
build-sandbox:
build-runtimes:
needs: compute-versions
permissions:
contents: read
uses: ./.github/workflows/build-sandbox-binaries.yml
uses: ./.github/workflows/build-runtime-binaries.yml
with:
cargo-version: ${{ needs.compute-versions.outputs.cargo_version }}
image-tag: dev
secrets: inherit
build-vm-driver:
needs: [compute-versions, build-sandbox]
needs: [compute-versions, build-runtimes]
permissions:
contents: read
uses: ./.github/workflows/build-vm-driver.yml
@@ -120,7 +120,7 @@ jobs:
secrets: inherit
package-binaries:
needs: [build-cli, build-gateway, build-sandbox, build-vm-driver]
needs: [build-cli, build-gateway, build-runtimes, build-vm-driver]
permissions:
actions: read
contents: read
@@ -138,20 +138,32 @@ jobs:
target-suffix: unknown-linux-gnu
secrets: inherit
build-sandbox-image:
needs: build-runtimes
permissions:
contents: read
packages: write
uses: ./.github/workflows/docker-build.yml
with:
component: sandbox
binary: openshell-sandbox
target-suffix: unknown-linux-musl
secrets: inherit
build-supervisor-image:
needs: build-sandbox
needs: build-runtimes
permissions:
contents: read
packages: write
uses: ./.github/workflows/docker-build.yml
with:
component: supervisor
binary: openshell-sandbox
target-suffix: unknown-linux-musl
binary: openshell-supervisor
target-suffix: unknown-linux-gnu
secrets: inherit
docker-e2e:
needs: [build-cli, build-conformance, build-gateway, build-supervisor-image]
needs: [build-cli, build-conformance, build-gateway, build-sandbox-image, build-supervisor-image]
permissions:
actions: read
contents: read
@@ -163,7 +175,7 @@ jobs:
conformance-artifact-prefix: openshell-conformance
podman-e2e:
needs: [build-cli, build-conformance, build-gateway, build-supervisor-image]
needs: [build-cli, build-conformance, build-gateway, build-sandbox-image, build-supervisor-image]
permissions:
actions: read
contents: read
@@ -185,7 +197,7 @@ jobs:
tag-ghcr-dev:
name: Tag GHCR Images as Dev
needs: [build-gateway-image, build-supervisor-image, release-dev]
needs: [build-gateway-image, build-sandbox-image, build-supervisor-image, release-dev]
runs-on: linux-amd64-cpu8
timeout-minutes: 10
steps:
@@ -196,7 +208,7 @@ jobs:
run: |
set -euo pipefail
REGISTRY="ghcr.io/nvidia/openshell"
for component in gateway supervisor; do
for component in gateway sandbox supervisor; do
echo "Tagging ${REGISTRY}/${component}:${{ github.sha }} as dev..."
docker buildx imagetools create \
--prefer-index=false \
@@ -253,7 +265,7 @@ jobs:
build-snap:
name: Build Snap
needs: [compute-versions, build-cli, build-gateway, build-sandbox]
needs: [compute-versions, build-cli, build-gateway, build-runtimes]
uses: ./.github/workflows/snap-package.yml
with:
checkout-ref: ${{ github.sha }}
@@ -281,6 +293,20 @@ jobs:
fail-fast: false
matrix:
include:
- name: ubuntu-20.04-binaries-amd64
runner: linux-amd64-cpu8
image: ubuntu:20.04
kind: binary
artifact_arch: amd64
target: x86_64-unknown-linux-gnu
sandbox_target: x86_64-unknown-linux-musl
- name: ubuntu-20.04-binaries-arm64
runner: linux-arm64-cpu8
image: ubuntu:20.04
kind: binary
artifact_arch: arm64
target: aarch64-unknown-linux-gnu
sandbox_target: aarch64-unknown-linux-musl
- name: ubuntu-22.04-deb-amd64
runner: linux-amd64-cpu8
image: ubuntu:22.04
@@ -313,6 +339,30 @@ jobs:
container:
image: ${{ matrix.image }}
steps:
- name: Download sandbox binary artifact
if: matrix.kind == 'binary'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: sandbox-binary-linux-${{ matrix.artifact_arch }}
path: runtime-input/
- name: Download supervisor binary artifact
if: matrix.kind == 'binary'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: supervisor-binary-linux-${{ matrix.artifact_arch }}
path: runtime-input/
- name: Smoke runtime binary artifacts
if: matrix.kind == 'binary'
run: |
set -euo pipefail
mkdir -p runtime-bin
tar -xzf "runtime-input/openshell-sandbox-${{ matrix.sandbox_target }}.tar.gz" -C runtime-bin
tar -xzf "runtime-input/openshell-supervisor-${{ matrix.target }}.tar.gz" -C runtime-bin
runtime-bin/openshell-sandbox --version
LD_BIND_NOW=1 runtime-bin/openshell-supervisor --version
- name: Download Debian package artifact
if: matrix.kind == 'deb'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
@@ -403,6 +453,13 @@ jobs:
path: release/
merge-multiple: true
- name: Download sandbox binary artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: sandbox-binary-*
path: release/
merge-multiple: true
- name: Download VM driver artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
@@ -479,7 +536,7 @@ jobs:
run: |
set -euo pipefail
cd release
sha256sum \
sha256sum -- \
openshell-x86_64-unknown-linux-musl.tar.gz \
openshell-aarch64-unknown-linux-musl.tar.gz \
openshell-aarch64-apple-darwin.tar.gz \
@@ -490,15 +547,19 @@ jobs:
openshell-*.rpm \
*.whl > openshell-checksums-sha256.txt
cat openshell-checksums-sha256.txt
sha256sum \
sha256sum -- \
openshell-gateway-x86_64-unknown-linux-gnu.tar.gz \
openshell-gateway-aarch64-unknown-linux-gnu.tar.gz \
openshell-gateway-aarch64-apple-darwin.tar.gz > openshell-gateway-checksums-sha256.txt
cat openshell-gateway-checksums-sha256.txt
sha256sum \
sha256sum -- \
openshell-sandbox-x86_64-unknown-linux-musl.tar.gz \
openshell-sandbox-aarch64-unknown-linux-musl.tar.gz > openshell-sandbox-checksums-sha256.txt
cat openshell-sandbox-checksums-sha256.txt
sha256sum -- \
openshell-supervisor-x86_64-unknown-linux-gnu.tar.gz \
openshell-supervisor-aarch64-unknown-linux-gnu.tar.gz > openshell-supervisor-checksums-sha256.txt
cat openshell-supervisor-checksums-sha256.txt
- name: Generate Homebrew formula
run: |
@@ -604,6 +665,8 @@ jobs:
release/openshell-gateway-aarch64-apple-darwin.tar.gz
release/openshell-sandbox-x86_64-unknown-linux-musl.tar.gz
release/openshell-sandbox-aarch64-unknown-linux-musl.tar.gz
release/openshell-supervisor-x86_64-unknown-linux-gnu.tar.gz
release/openshell-supervisor-aarch64-unknown-linux-gnu.tar.gz
release/openshell-driver-vm-x86_64-unknown-linux-gnu.tar.gz
release/openshell-driver-vm-aarch64-unknown-linux-gnu.tar.gz
release/openshell-driver-vm-aarch64-apple-darwin.tar.gz
@@ -612,6 +675,7 @@ jobs:
release/openshell-checksums-sha256.txt
release/openshell-gateway-checksums-sha256.txt
release/openshell-sandbox-checksums-sha256.txt
release/openshell-supervisor-checksums-sha256.txt
release-helm:
name: Release Helm Chart (OCI, dev)
.github/workflows/release-tag.yml
+65 -15
View File
@@ -129,11 +129,11 @@ jobs:
checkout-ref: ${{ inputs.tag || github.ref }}
secrets: inherit
build-sandbox:
build-runtimes:
needs: compute-versions
permissions:
contents: read
uses: ./.github/workflows/build-sandbox-binaries.yml
uses: ./.github/workflows/build-runtime-binaries.yml
with:
cargo-version: ${{ needs.compute-versions.outputs.cargo_version }}
image-tag: ${{ needs.compute-versions.outputs.semver }}
@@ -141,7 +141,7 @@ jobs:
secrets: inherit
build-vm-driver:
needs: [compute-versions, build-sandbox]
needs: [compute-versions, build-runtimes]
permissions:
contents: read
uses: ./.github/workflows/build-vm-driver.yml
@@ -152,7 +152,7 @@ jobs:
secrets: inherit
package-binaries:
needs: [build-cli, build-gateway, build-sandbox, build-vm-driver]
needs: [build-cli, build-gateway, build-runtimes, build-vm-driver]
permissions:
actions: read
contents: read
@@ -172,22 +172,36 @@ jobs:
checkout-ref: ${{ inputs.tag || github.ref }}
secrets: inherit
build-supervisor-image:
needs: [compute-versions, build-sandbox]
build-sandbox-image:
needs: [compute-versions, build-runtimes]
permissions:
contents: read
packages: write
uses: ./.github/workflows/docker-build.yml
with:
component: supervisor
component: sandbox
binary: openshell-sandbox
target-suffix: unknown-linux-musl
image-tag: ${{ needs.compute-versions.outputs.source_sha }}
checkout-ref: ${{ inputs.tag || github.ref }}
secrets: inherit
build-supervisor-image:
needs: [compute-versions, build-runtimes]
permissions:
contents: read
packages: write
uses: ./.github/workflows/docker-build.yml
with:
component: supervisor
binary: openshell-supervisor
target-suffix: unknown-linux-gnu
image-tag: ${{ needs.compute-versions.outputs.source_sha }}
checkout-ref: ${{ inputs.tag || github.ref }}
secrets: inherit
docker-e2e:
needs: [compute-versions, build-cli, build-conformance, build-gateway, build-supervisor-image]
needs: [compute-versions, build-cli, build-conformance, build-gateway, build-sandbox-image, build-supervisor-image]
permissions:
actions: read
contents: read
@@ -200,7 +214,7 @@ jobs:
conformance-artifact-prefix: openshell-conformance
podman-e2e:
needs: [compute-versions, build-cli, build-conformance, build-gateway, build-supervisor-image]
needs: [compute-versions, build-cli, build-conformance, build-gateway, build-sandbox-image, build-supervisor-image]
permissions:
actions: read
contents: read
@@ -224,7 +238,7 @@ jobs:
tag-ghcr-release:
name: Tag GHCR Images
needs: [compute-versions, build-gateway-image, build-supervisor-image, release]
needs: [compute-versions, build-gateway-image, build-sandbox-image, build-supervisor-image, release]
runs-on: linux-amd64-cpu8
timeout-minutes: 10
steps:
@@ -239,7 +253,7 @@ jobs:
REGISTRY="ghcr.io/nvidia/openshell"
VERSION="${{ needs.compute-versions.outputs.semver }}"
SOURCE_TAG="${{ needs.compute-versions.outputs.source_sha }}"
for component in gateway supervisor; do
for component in gateway sandbox supervisor; do
echo "Tagging ${REGISTRY}/${component}:${SOURCE_TAG} as ${VERSION}..."
docker buildx imagetools create \
--prefer-index=false \
@@ -303,7 +317,7 @@ jobs:
build-snap:
name: Build Snap
needs: [compute-versions, build-cli, build-gateway, build-sandbox]
needs: [compute-versions, build-cli, build-gateway, build-runtimes]
uses: ./.github/workflows/snap-package.yml
with:
checkout-ref: ${{ inputs.tag || github.ref }}
@@ -339,6 +353,7 @@ jobs:
artifact_arch: amd64
rpm_arch: x86_64
target: x86_64-unknown-linux-gnu
sandbox_target: x86_64-unknown-linux-musl
- name: ubuntu-20.04-binaries-arm64
runner: linux-arm64-cpu8
image: ubuntu:20.04
@@ -346,6 +361,7 @@ jobs:
artifact_arch: arm64
rpm_arch: aarch64
target: aarch64-unknown-linux-gnu
sandbox_target: aarch64-unknown-linux-musl
- name: ubuntu-22.04-deb
runner: linux-amd64-cpu8
image: ubuntu:22.04
@@ -396,6 +412,20 @@ jobs:
name: driver-vm-linux-${{ matrix.artifact_arch }}
path: smoke-input/
- name: Download sandbox binary artifact
if: matrix.kind == 'binary'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: sandbox-binary-linux-${{ matrix.artifact_arch }}
path: smoke-input/
- name: Download supervisor binary artifact
if: matrix.kind == 'binary'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: supervisor-binary-linux-${{ matrix.artifact_arch }}
path: smoke-input/
- name: Smoke binary artifacts
if: matrix.kind == 'binary'
run: |
@@ -404,8 +434,12 @@ jobs:
mkdir -p smoke-bin
tar -xzf smoke-input/openshell-gateway-${{ matrix.target }}.tar.gz -C smoke-bin
tar -xzf smoke-input/openshell-driver-vm-${{ matrix.target }}.tar.gz -C smoke-bin
tar -xzf smoke-input/openshell-sandbox-${{ matrix.sandbox_target }}.tar.gz -C smoke-bin
tar -xzf smoke-input/openshell-supervisor-${{ matrix.target }}.tar.gz -C smoke-bin
LD_BIND_NOW=1 smoke-bin/openshell-gateway --version
smoke-bin/openshell-driver-vm --version
smoke-bin/openshell-sandbox --version
LD_BIND_NOW=1 smoke-bin/openshell-supervisor --version
- name: Download Debian package artifact
if: matrix.kind == 'deb'
@@ -499,6 +533,13 @@ jobs:
path: release/
merge-multiple: true
- name: Download sandbox binary artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: sandbox-binary-*
path: release/
merge-multiple: true
- name: Download VM driver artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
@@ -550,7 +591,7 @@ jobs:
run: |
set -euo pipefail
cd release
sha256sum \
sha256sum -- \
openshell-x86_64-unknown-linux-musl.tar.gz \
openshell-aarch64-unknown-linux-musl.tar.gz \
openshell-aarch64-apple-darwin.tar.gz \
@@ -561,15 +602,19 @@ jobs:
openshell-*.rpm \
*.whl > openshell-checksums-sha256.txt
cat openshell-checksums-sha256.txt
sha256sum \
sha256sum -- \
openshell-gateway-x86_64-unknown-linux-gnu.tar.gz \
openshell-gateway-aarch64-unknown-linux-gnu.tar.gz \
openshell-gateway-aarch64-apple-darwin.tar.gz > openshell-gateway-checksums-sha256.txt
cat openshell-gateway-checksums-sha256.txt
sha256sum \
sha256sum -- \
openshell-sandbox-x86_64-unknown-linux-musl.tar.gz \
openshell-sandbox-aarch64-unknown-linux-musl.tar.gz > openshell-sandbox-checksums-sha256.txt
cat openshell-sandbox-checksums-sha256.txt
sha256sum -- \
openshell-supervisor-x86_64-unknown-linux-gnu.tar.gz \
openshell-supervisor-aarch64-unknown-linux-gnu.tar.gz > openshell-supervisor-checksums-sha256.txt
cat openshell-supervisor-checksums-sha256.txt
- name: Generate Homebrew formula
if: needs.compute-versions.outputs.is_prerelease != 'true'
@@ -642,6 +687,8 @@ jobs:
release/openshell-gateway-aarch64-apple-darwin.tar.gz
release/openshell-sandbox-x86_64-unknown-linux-musl.tar.gz
release/openshell-sandbox-aarch64-unknown-linux-musl.tar.gz
release/openshell-supervisor-x86_64-unknown-linux-gnu.tar.gz
release/openshell-supervisor-aarch64-unknown-linux-gnu.tar.gz
release/openshell-driver-vm-x86_64-unknown-linux-gnu.tar.gz
release/openshell-driver-vm-aarch64-unknown-linux-gnu.tar.gz
release/openshell-driver-vm-aarch64-apple-darwin.tar.gz
@@ -650,6 +697,7 @@ jobs:
release/openshell-checksums-sha256.txt
release/openshell-gateway-checksums-sha256.txt
release/openshell-sandbox-checksums-sha256.txt
release/openshell-supervisor-checksums-sha256.txt
- name: Upload pre-release artifacts
if: needs.compute-versions.outputs.is_prerelease == 'true'
@@ -706,6 +754,8 @@ jobs:
run: |
{
echo "@nvidia:registry=https://npm.pkg.github.com"
# Keep the token reference literal for npm to expand at publish time.
# shellcheck disable=SC2016
echo '//npm.pkg.github.com/:_authToken=${NODE_AUTH_TOKEN}'
} > .npmrc
.github/workflows/release-vm-kernel.yml
+11 -6
View File
@@ -1,6 +1,6 @@
name: Release VM Kernel
# Build custom libkrunfw (kernel firmware) + libkrun (VMM) + gvproxy for all
# Build custom libkrunfw (kernel firmware) + libkrun (VMM) for all
# supported openshell-driver-vm platforms. Artifacts are uploaded to the
# rolling "vm-runtime" GitHub Release and consumed by normal dev/tag release
# workflows when building the openshell-driver-vm binary.
@@ -194,23 +194,28 @@ jobs:
release/vm-runtime-darwin-aarch64.tar.zst
- name: Ensure vm-runtime tag exists
env:
RELEASE_TAG: vm-runtime
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git tag -fa vm-runtime -m "VM Runtime Development Build" "${GITHUB_SHA}"
git push --force origin vm-runtime
git tag -fa "$RELEASE_TAG" -m "VM Runtime Development Build" "${GITHUB_SHA}"
git push --force origin "$RELEASE_TAG"
- name: Prune stale runtime assets from vm-runtime release
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
env:
RELEASE_TAG: vm-runtime
with:
script: |
const [owner, repo] = process.env.GITHUB_REPOSITORY.split('/');
const tag = process.env.RELEASE_TAG;
let release;
try {
release = await github.rest.repos.getReleaseByTag({ owner, repo, tag: 'vm-runtime' });
release = await github.rest.repos.getReleaseByTag({ owner, repo, tag });
} catch (err) {
if (err.status === 404) {
core.info('No existing vm-runtime release; will create fresh.');
core.info(`No existing ${tag} release; will create fresh.`);
return;
}
throw err;
@@ -237,7 +242,7 @@ jobs:
### Kernel Runtime Artifacts
Pre-built kernel runtime (libkrunfw + libkrun + gvproxy + umoci) for embedding
Pre-built kernel runtime (libkrunfw + libkrun + umoci) for embedding
into the `openshell-driver-vm` binary. These are rebuilt on demand when the
kernel config or pinned dependency versions change.
.github/workflows/windows-msvc.yml
+3
View File
@@ -56,6 +56,7 @@ jobs:
rust_target: aarch64-pc-windows-msvc
runs-on: ${{ matrix.runner }}
env:
READ_ONLY_GITHUB_TOKEN: ${{ github.token }}
SCCACHE_GHA_ENABLED: "true"
SCCACHE_GHA_VERSION: windows-msvc-${{ matrix.arch }}
SCCACHE_CLIENT_SIDE: "1"
@@ -121,6 +122,7 @@ jobs:
rust_target: aarch64-pc-windows-msvc
runs-on: ${{ matrix.runner }}
env:
READ_ONLY_GITHUB_TOKEN: ${{ github.token }}
SCCACHE_GHA_ENABLED: "true"
SCCACHE_GHA_VERSION: windows-msvc-${{ matrix.arch }}
SCCACHE_CLIENT_SIDE: "1"
@@ -178,6 +180,7 @@ jobs:
rust_target: aarch64-pc-windows-msvc
runs-on: ${{ matrix.runner }}
env:
READ_ONLY_GITHUB_TOKEN: ${{ github.token }}
SCCACHE_GHA_ENABLED: "true"
SCCACHE_GHA_VERSION: windows-msvc-${{ matrix.arch }}
SCCACHE_CLIENT_SIDE: "1"
.gitignore
+3
View File
@@ -186,6 +186,9 @@ temp/
*.tmp
e2e/gpu/images/.build/
# Ansible Galaxy roles
/tests/ansible/.roles/
# Secrets/credentials (should never be committed)
*.pem
*.key
AGENTS.md
+5 -1
View File
@@ -38,7 +38,11 @@ These pipelines connect skills into end-to-end workflows. Individual skill files
| `crates/openshell-conformance/` | CLI conformance library | Reusable driver-agnostic scenarios and command runner |
| `crates/openshell-conformance-cli/` | Conformance CLI | Distributable `list` and `run` entrypoint for gateway conformance |
| `crates/openshell-server/` | Gateway server | Control-plane API, sandbox lifecycle, auth boundary |
| `crates/openshell-sandbox/` | Sandbox runtime | Container supervision, policy-enforced egress routing |
| `crates/openshell-sandbox/` | Sandbox runtime | Capability-free workload launcher, process identity, and seccomp-mediated I/O |
| `crates/openshell-supervisor/` | Supervisor runtime | Gateway session, policy evaluation, credentials, and upstream networking |
| `crates/openshell-binary-identity/` | Binary identity | Shared trusted procfs executable identity resolution for isolation backends |
| `crates/openshell-isolation-interface/` | Isolation backend interface | RFC 0012 `IsolationBackend` trait and types; the supervisor-facing runtime contract |
| `crates/openshell-sandbox-backend/` | OpenShell sandbox backend | `OpenShellRuntimeBackend` and the authenticated OpenShell Sandbox Protocol shared with `openshell-sandbox` |
| `crates/openshell-policy/` | Policy engine | Filesystem, network, and process constraints |
| `crates/openshell-bootstrap/` | Gateway metadata | Gateway registration metadata, auth token storage, mTLS bundle storage |
| `crates/openshell-gateway-interceptors/` | Gateway interceptors | Intercepts and transforms configured gRPC requests at the gateway routing boundary |
CI.md
+2 -2
View File
@@ -373,9 +373,9 @@ These workflows run after merge to publish dev/tagged artifacts and verify them.
| File | Role |
|---|---|
| `.github/workflows/release-dev.yml` | Publishes the rolling `dev` build on every push to `main`. Builds gateway/supervisor images and binaries, packages, wheels, and pushes the Helm chart as `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` (plus an immutable `0.0.0-dev.<sha>` pin). Also dispatchable manually. |
| `.github/workflows/release-dev.yml` | Publishes the rolling `dev` build on every push to `main`. Builds gateway, sandbox, and supervisor images and binaries, packages, wheels, and pushes the Helm chart as `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` (plus an immutable `0.0.0-dev.<sha>` pin). Also dispatchable manually. |
| `.github/workflows/release-tag.yml` | Publishes a tagged stable release. Its automatic tag trigger excludes `-pre.*`; manual dispatch remains maintainer-controlled. |
| `.github/workflows/release-canary.yml` | Smoke-tests published artifacts on `macos`, `ubuntu`, `fedora`, and `kubernetes` (kind + Helm) runners. Triggers automatically when `Release Dev` succeeds, and via `workflow_dispatch` on any branch (`gh workflow run release-canary.yml --ref <branch>`). The `kubernetes` job pins to `0.0.0-dev` artifacts; the other jobs install the latest tagged release via `install.sh`. See the `test-release-canary` skill for the manual-dispatch playbook and local kind reproduction. |
| `.github/workflows/release-canary.yml` | Smoke-tests published dev artifacts on `macos`, `ubuntu`, `fedora`, and `kubernetes` (kind + Helm) runners. Each job reaches its gateway and creates, exercises, and deletes a sandbox. It runs automatically after `Release Dev` succeeds and supports manual dispatch (`gh workflow run release-canary.yml --ref <branch>`). See the `test-release-canary` skill for the playbook and local kind reproduction. |
## Required status contexts
CONTRIBUTING.md
+17 -12
View File
@@ -340,14 +340,19 @@ cargo build -p openshell-prover --features bundled-z3
For x86-64 and ARM64 Windows MSVC builds, use one of these Z3 paths:
- Bundled Z3 (the default for `windows:*` tasks): `z3-sys` builds Z3 from
source and links it into the OpenShell binaries. This keeps the release
executables independent of `libz3.dll` when they are copied to another host.
- Prebuilt Z3 (the default for `windows:*` tasks): `z3-sys` downloads the
pinned Z3 4.16.0 GitHub release for the target architecture on the first
build. Cargo reuses the extracted archive from its target directory. Windows
CI authenticates the GitHub API request with `READ_ONLY_GITHUB_TOKEN` and
preserves the archive in the architecture-specific Cargo target cache. For
cold local builds, you may set `READ_ONLY_GITHUB_TOKEN` to avoid anonymous
GitHub API rate limits.
- System Z3: point `Z3_LIBRARY_PATH_OVERRIDE` at the directory containing the
target-compatible MSVC Z3 library and `Z3_SYS_Z3_HEADER` at the full path to
`z3.h`. The `windows:*` tasks use this path automatically when
`Z3_LIBRARY_PATH_OVERRIDE` is set. Deploy the matching `libz3.dll` with any
binaries built against a dynamic system Z3.
target-compatible MSVC Z3 library and `Z3_SYS_Z3_HEADER` at the full path to `z3.h`.
The `windows:*` tasks use this path automatically when `Z3_LIBRARY_PATH_OVERRIDE`
is set.
- Bundled Z3: for direct Cargo builds, pass `--features bundled-z3` so `z3-sys`
builds Z3 from source.
`openshell-prover` itself has no `bindgen`/`libclang` dependency, so building
just this crate does not require `LIBCLANG_PATH`:
@@ -360,17 +365,17 @@ cargo build -p openshell-prover --target x86_64-pc-windows-msvc --features bundl
To build the full set of Windows binaries, including `openshell-gateway.exe`
and `openshell.exe`, use the `windows:build:x64` mise task instead of a
single-crate `cargo build`. It builds Z3 from source (bundled) by default. A full
build also compiles crates that use `bindgen` (e.g. the MXC driver on Windows),
so it requires `libclang.dll`; if LLVM is not on the default search path, set
`LIBCLANG_PATH` to the directory containing `libclang.dll`:
single-crate `cargo build`. It downloads the pinned prebuilt Z3 release by default. A
full build also compiles crates that use `bindgen` (e.g. the MXC driver on
Windows), so it requires `libclang.dll`; if LLVM is not on the default search
path, set `LIBCLANG_PATH` to the directory containing `libclang.dll`:
```powershell
$env:LIBCLANG_PATH='C:\Program Files\Microsoft Visual Studio\2022\<Edition>\VC\Tools\Llvm\x64\bin'
mise run --skip-tools windows:build:x64
```
To use a local x64 Z3 release instead of the bundled build, set
To use a local x64 Z3 release instead of the prebuilt download, set
`Z3_LIBRARY_PATH_OVERRIDE` and `Z3_SYS_Z3_HEADER` before running the task:
```powershell
Cargo.lock
Generated
+138 -19
View File
@@ -3935,6 +3935,14 @@ version = "1.70.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"
[[package]]
name = "openshell-binary-identity"
version = "0.0.0"
dependencies = [
"openshell-isolation-interface",
"sha2 0.10.9",
]
[[package]]
name = "openshell-bootstrap"
version = "0.0.0"
@@ -4035,6 +4043,7 @@ dependencies = [
"chrono",
"glob",
"ipnet",
"jsonwebtoken",
"miette",
"nix 0.29.0",
"openshell-extension-core",
@@ -4057,8 +4066,11 @@ dependencies = [
"tonic",
"tonic-prost",
"tonic-prost-build",
"tonic-types",
"tracing",
"url",
"uuid",
"zeroize",
]
[[package]]
@@ -4088,15 +4100,21 @@ dependencies = [
"clap",
"futures",
"http 1.4.0",
"libc",
"miette",
"openshell-core",
"openshell-isolation-interface",
"openshell-otel",
"openshell-otel-test-support",
"openshell-sandbox-backend",
"opentelemetry",
"opentelemetry_sdk",
"prost-types",
"rand 0.9.4",
"rustix 1.1.4",
"serde",
"serde_json",
"sha2 0.10.9",
"tar",
"temp-env",
"tempfile",
@@ -4126,13 +4144,17 @@ dependencies = [
"miette",
"notify",
"openshell-core",
"openshell-isolation-interface",
"openshell-otel",
"openshell-otel-test-support",
"openshell-policy",
"openshell-sandbox-backend",
"opentelemetry",
"opentelemetry_sdk",
"prost",
"prost-types",
"rand 0.9.4",
"rcgen",
"serde",
"serde_json",
"temp-env",
@@ -4202,14 +4224,17 @@ dependencies = [
"miette",
"nix 0.29.0",
"openshell-core",
"openshell-isolation-interface",
"openshell-otel",
"openshell-otel-test-support",
"openshell-sandbox-backend",
"opentelemetry",
"opentelemetry_sdk",
"prost-types",
"rustix 1.1.4",
"serde",
"serde_json",
"tar",
"temp-env",
"thiserror 2.0.18",
"tokio",
@@ -4220,6 +4245,7 @@ dependencies = [
"tracing-opentelemetry",
"tracing-subscriber",
"url",
"uuid",
]
[[package]]
@@ -4230,6 +4256,7 @@ dependencies = [
"futures",
"miette",
"openshell-core",
"rcgen",
"reqwest 0.12.28",
"rustls",
"serde",
@@ -4237,10 +4264,12 @@ dependencies = [
"sha2 0.10.9",
"tempfile",
"tokio",
"tokio-rustls",
"toml",
"tonic",
"tracing",
"tracing-subscriber",
"url",
"wiremock",
]
@@ -4248,6 +4277,7 @@ dependencies = [
name = "openshell-driver-vm"
version = "0.0.0"
dependencies = [
"base64",
"bollard",
"clap",
"flate2",
@@ -4260,15 +4290,18 @@ dependencies = [
"oci-client",
"openshell-core",
"openshell-driver-podman",
"openshell-isolation-interface",
"openshell-otel",
"openshell-otel-test-support",
"openshell-policy",
"openshell-sandbox-backend",
"openshell-vfio",
"opentelemetry",
"opentelemetry_sdk",
"polling",
"prost",
"prost-types",
"rand 0.9.4",
"rustix 1.1.4",
"serde",
"serde_json",
@@ -4350,6 +4383,18 @@ dependencies = [
"tracing-subscriber",
]
[[package]]
name = "openshell-isolation-interface"
version = "0.0.0"
dependencies = [
"async-trait",
"libc",
"openshell-core",
"rustix 1.1.4",
"serde",
"tokio",
]
[[package]]
name = "openshell-ocsf"
version = "0.0.0"
@@ -4438,31 +4483,71 @@ dependencies = [
name = "openshell-sandbox"
version = "0.0.0"
dependencies = [
"anyhow",
"async-trait",
"base64",
"bytes",
"capctl",
"clap",
"futures",
"hex",
"ipnet",
"landlock",
"libc",
"miette",
"nix 0.29.0",
"openshell-binary-identity",
"openshell-core",
"openshell-extension-core",
"openshell-isolation-interface",
"openshell-ocsf",
"openshell-policy",
"openshell-supervisor-middleware",
"openshell-supervisor-middleware-builtins",
"openshell-supervisor-network",
"openshell-supervisor-process",
"prost",
"prost-types",
"openshell-sandbox-backend",
"rand 0.10.2",
"rcgen",
"rustix 1.1.4",
"rustls",
"rustls-pemfile",
"seccompiler",
"serde",
"serde_json",
"temp-env",
"sha2 0.10.9",
"socket2",
"tempfile",
"tokio",
"tokio-tungstenite 0.26.2",
"tokio-rustls",
"tokio-stream",
"tonic",
"tracing",
"tracing-appender",
"tracing-subscriber",
]
[[package]]
name = "openshell-sandbox-backend"
version = "0.0.0"
dependencies = [
"async-trait",
"hyper-util",
"libc",
"openshell-core",
"openshell-isolation-interface",
"prost",
"protoc-bin-vendored",
"rcgen",
"rustix 1.1.4",
"rustls",
"rustls-pemfile",
"serde",
"serde_json",
"sha2 0.10.9",
"socket2",
"thiserror 2.0.18",
"tokio",
"tokio-rustls",
"tokio-stream",
"tonic",
"tonic-prost",
"tonic-prost-build",
"tower",
"tracing",
"uuid",
]
@@ -4594,6 +4679,41 @@ dependencies = [
"syn 2.0.117",
]
[[package]]
name = "openshell-supervisor"
version = "0.0.0"
dependencies = [
"clap",
"futures",
"miette",
"nix 0.29.0",
"openshell-core",
"openshell-extension-core",
"openshell-isolation-interface",
"openshell-ocsf",
"openshell-policy",
"openshell-sandbox-backend",
"openshell-supervisor-middleware",
"openshell-supervisor-middleware-builtins",
"openshell-supervisor-network",
"openshell-supervisor-process",
"prost",
"prost-types",
"rustix 1.1.4",
"rustls",
"serde",
"serde_json",
"temp-env",
"tempfile",
"tokio",
"tokio-tungstenite 0.26.2",
"tonic",
"tracing",
"tracing-appender",
"tracing-subscriber",
"uuid",
]
[[package]]
name = "openshell-supervisor-middleware"
version = "0.0.0"
@@ -4631,6 +4751,7 @@ name = "openshell-supervisor-network"
version = "0.0.0"
dependencies = [
"apollo-parser",
"async-trait",
"aws-credential-types",
"aws-sigv4",
"aws-smithy-runtime-api",
@@ -4646,7 +4767,9 @@ dependencies = [
"libc",
"miette",
"noyalib",
"openshell-binary-identity",
"openshell-core",
"openshell-isolation-interface",
"openshell-ocsf",
"openshell-policy",
"openshell-supervisor-middleware",
@@ -4683,25 +4806,20 @@ name = "openshell-supervisor-process"
version = "0.0.0"
dependencies = [
"anyhow",
"async-trait",
"base64",
"bytes",
"capctl",
"hex",
"ipnet",
"landlock",
"libc",
"miette",
"nix 0.29.0",
"openshell-core",
"openshell-isolation-interface",
"openshell-ocsf",
"openshell-policy",
"rand 0.10.2",
"russh",
"rustix 1.1.4",
"seccompiler",
"serde_json",
"sha2 0.10.9",
"socket2",
"tempfile",
"tokio",
"tokio-stream",
@@ -5726,6 +5844,7 @@ dependencies = [
"pem",
"rustls-pki-types",
"time",
"x509-parser",
"yasna",
]
@@ -7176,7 +7295,6 @@ version = "0.3.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "96374855068f47402c3121c6eed88d29cb1de8f3ab27090e273e420bdabcf050"
dependencies = [
"futures",
"parking_lot",
]
@@ -8747,6 +8865,7 @@ dependencies = [
"lazy_static",
"nom",
"oid-registry",
"ring",
"rusticata-macros",
"thiserror 1.0.69",
"time",
Cargo.toml
+4 -1
View File
@@ -18,6 +18,7 @@ tokio = { version = "1.43", features = ["full"] }
# gRPC/Protobuf
tonic = "0.14"
tonic-types = "0.14"
tonic-prost = "0.14"
tonic-prost-build = "0.14"
prost = "0.14"
@@ -33,12 +34,13 @@ hyper-util = { version = "0.1", features = ["tokio", "server-auto"] }
http = "1.2"
http-body = "1.0"
http-body-util = "0.1"
h2 = "0.4"
# TLS
tokio-rustls = { version = "0.26", default-features = false, features = ["logging", "tls12", "aws_lc_rs"] }
rustls = { version = "0.23", default-features = false, features = ["std", "logging", "tls12", "aws_lc_rs"] }
rustls-pemfile = "2"
rcgen = { version = "0.13", default-features = false, features = ["crypto", "pem", "aws_lc_rs"] }
rcgen = { version = "0.13", default-features = false, features = ["crypto", "pem", "aws_lc_rs", "x509-parser"] }
webpki-roots = "1"
rustls-native-certs = "0.8"
@@ -106,6 +108,7 @@ base64 = "0.22"
sha2 = "0.10"
rand = "0.9"
jsonwebtoken = { version = "10", features = ["aws_lc_rs"] }
zeroize = { version = "1", features = ["derive"] }
getrandom = "0.3"
aws-lc-rs = "1.16"
spiffe = { version = "0.15", default-features = false, features = ["workload-api-jwt", "jwt-verify-rust-crypto", "tracing"] }
README.md
+2 -1
View File
@@ -48,7 +48,8 @@ uv add openshell
Deploy the OpenShell gateway into a Kubernetes cluster from the OCI chart published to GHCR:
```bash
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true
```
See [`deploy/helm/openshell/README.md`](deploy/helm/openshell/README.md) for available versions, dev tag conventions, and configuration.
TESTING.md
+18
View File
@@ -48,6 +48,24 @@ mise run test:rust # cargo test --workspace
Rust validation checks tracked Cargo lockfiles; run `mise run rust:lockfiles:check` to check them directly. If one is stale, refresh it with Cargo using its adjacent manifest, review the diff, and commit the update.
### Native Windows validation
Use `mise run --skip-tools pre-commit` with the existing Rust/MSVC toolchain.
Windows now checks tracked Cargo lockfiles through PowerShell rather than
skipping them. The deterministic gateway parity task uses Git for Windows Bash,
with temporary Python launchers confined to a unique checkout-owned directory.
`mise run --skip-tools sdk:ts:ci` selects the x64 Biome executable on Windows
(including ARM64 hosts running it under emulation), resolves the protobuf
plugin through its Windows `.cmd` launcher, and installs the matching locked
ARM64 Rolldown binding when Node itself is ARM64. The helper preserves lockfile
resolution; it must not upgrade unrelated test dependencies.
`mise run --skip-tools go:ci` retains race detection except on Windows ARM64,
where Go does not support it. Windows token-file tests explicitly skip POSIX
mode-bit assertions; those skips do not establish Windows ACL protection.
Use a checkout with LF text files when running Unix-shell fixture checks.
## Python Unit Tests
Python unit tests use the `*_test.py` suffix convention (not `test_*` prefix)
architecture/build.md
+71 -148
View File
@@ -16,7 +16,8 @@ OpenShell builds these main artifacts:
| Python SDK wheel | `python/openshell` |
| TypeScript SDK package | `sdk/typescript` |
| Gateway container image | `deploy/docker/Dockerfile.gateway` |
| Supervisor container image | `deploy/docker/Dockerfile.supervisor` |
| Sandbox runtime binary and container image | `crates/openshell-sandbox` and `deploy/docker/Dockerfile.sandbox` |
| Supervisor binary and container image | `crates/openshell-supervisor` and `deploy/docker/Dockerfile.supervisor` |
| Helm chart | `deploy/helm/openshell` |
| VM driver/runtime assets | `crates/openshell-driver-vm` |
| Published docs site | `docs/` rendered by Fern config in `fern/` |
@@ -25,21 +26,11 @@ Sandbox community images are built outside this repository.
## Build Features
Rust builds require Rust 1.94 or newer. TLS and certificate generation use
AWS-LC, including the CLI and standalone examples. Native and cross-build
environments must provide the C toolchain required by aws-lc-sys; the Nix
development shells provide static AWS-LC libraries.
SQLx uses AWS-LC with native certificate roots. The server enables
`sqlx-core/rustls-native-certs` directly because SQLx's facade does not expose
that root selection independently of the crypto provider. Credential storage
continues to use the same AES-256-GCM envelope format across backend changes.
Anonymous telemetry emission is gated behind a default-on `telemetry` Cargo
feature. It is defined in `openshell-core` (where the emission code, HTTP
client, and endpoint live) and forwarded by the binary crates that emit or
collect telemetry: `openshell-gateway`, `openshell-sandbox`
(supervisor), and `openshell-driver-vm`. Every crate depends on
collect telemetry: `openshell-gateway`, `openshell-sandbox`,
`openshell-supervisor`, and `openshell-driver-vm`. Every crate depends on
`openshell-core` with `default-features = false`, so the binary crate's feature
is the single switch that enables `openshell-core/telemetry` for its build
graph. In-process drivers (`docker`, `kubernetes`, `podman`) inherit the
@@ -57,8 +48,7 @@ Cargo cannot subtract a single default feature, so each of the three binary
crates also defines a `defaults-without-telemetry` alias listing every default
except `telemetry`. Telemetry-free builds use
`--no-default-features --features defaults-without-telemetry` and stay correct
as the default set grows, instead of dropping unrelated defaults the way a bare
`--no-default-features` does on `openshell-sandbox`. The alias is a keep-list,
as the default set grows. The alias is a keep-list,
not a switch: enabling it on top of the defaults would otherwise yield a
telemetry-on binary that reads as telemetry-free, so each crate root carries a
`compile_error!` for the `telemetry` + `defaults-without-telemetry` combination.
@@ -73,7 +63,7 @@ roots through `webpki-roots` plus locally-installed CAs from the system bundle.
Building without `bundled-ca-roots` switches to the platform trust store via
`rustls-native-certs` and excludes bundled Mozilla root crates such as
`webpki-roots` and `webpki-root-certs` from the dependency graph. The
`system-ca-roots` feature alias on `openshell-sandbox` includes all other
`system-ca-roots` feature alias on `openshell-supervisor` includes all other
defaults (currently `telemetry`) except `bundled-ca-roots`, so Linux
distribution builds (e.g. RPM) can use
`--no-default-features --features system-ca-roots` without manually re-adding
@@ -96,38 +86,19 @@ The gateway bundles z3 into the release binary so Linux packages, standalone
tarballs, and gateway images do not depend on distro-specific z3 shared-library
SONAMEs.
The supervisor is the one binary whose libc is selectable, because it is the one
binary executed inside a userland OpenShell does not control. `SUPERVISOR_LIBC`
chooses between `musl` (default) and `glibc-static`. Both produce a fully static
binary; the choice does not change the runtime layout or the supervisor image base.
Static linkage is a hard requirement rather than a preference, so both variants
are verified by `tasks/scripts/verify-static-binary.sh`, which fails the build on
any `PT_INTERP` or `DT_NEEDED` entry.
The two variants differ only in build-time constraints:
| | `musl` (default) | `glibc-static` |
|---|---|---|
| Cross-compiles | yes, via `cargo zigbuild` | no — must build natively per architecture |
| Host requirement | zig + cargo-zigbuild | glibc static libraries (`glibc-static` on Fedora/RHEL, `libc6-dev` on Debian/Ubuntu) |
| libc license | MIT | LGPL-2.1-or-later, statically linked |
`cargo zigbuild` cannot produce the `glibc-static` variant: `zig cc` accepts
`-static` for `*-linux-gnu` targets and emits a dynamically linked binary
anyway. The staging script therefore refuses to cross-compile that variant
instead of silently degrading linkage.
Selecting `glibc-static` statically links LGPL glibc into a redistributed
binary, which carries relinking obligations that musl (MIT) does not. Treat the
default as the shipping configuration unless that has been reviewed.
The workload-side `openshell-sandbox` binary is statically linked with musl so
drivers can stage it into an arbitrary agent image without depending on that
image's libc. The supervisor is also statically linked: release images use the
default musl build, while distribution-specific builds may select the
`glibc-static` variant.
## Container Builds
The Docker image pipeline is a two-step flow: build the Rust binary natively
for the target architecture, then assemble the container image from the
prebuilt binary. The gateway image is built from `deploy/docker/Dockerfile.gateway`
and the supervisor image from `deploy/docker/Dockerfile.supervisor`. Neither
Dockerfile compiles Rust — both copy a staged binary out of
prebuilt binary. The gateway, sandbox, and supervisor images use distinct
Dockerfiles under `deploy/docker/`. None of the Dockerfiles compile Rust; they
copy staged binaries out of
`deploy/docker/.build/prebuilt-binaries/<arch>/` into the final image.
Local binary staging is driven by `tasks/scripts/stage-prebuilt-binaries.sh`. Because
@@ -157,28 +128,6 @@ in platform-specific Nix development shells through reusable workflows and the
shared `build-rust-binary` action. The image build downloads each binary artifact
into the staging directory before running Buildx.
The Nix flake exposes one development shell with target-specific toolchains.
The shared `mkToolchain` function in `nix/toolchain/default.nix` assembles native
libraries and Cargo environment settings. Linux and Darwin modules select the
compiler and sysroot and generate the compiler wrapper for their platform.
The `nix/toolchain/glibc-2.28/` directory contains the pinned glibc build,
GCC environment, and sysroot assembly used by GNU Linux targets.
The glibc build reuses a pinned historical Nixpkgs recipe with current build
tools; its headers, shared libraries, and static archives are assembled into
the sysroot from separate outputs.
Each toolchain supplies its compiler driver, assembler, archiver, native
libraries, and Cargo environment through derivation passthru. The shell omits
an implicit host C compiler; Cargo builds select the appropriate tools with
`--target`. GNU targets use a glibc 2.28 sysroot and static GCC runtimes, while
musl targets produce static executables.
On macOS, the shell also provides a native Darwin toolchain with static Z3
and AWS-LC. Its Clang driver uses the pinned, unprocessed Apple SDK so system
library stubs, including libiconv and libc++, retain their Apple install names.
System libraries and frameworks remain dynamically linked. The deployment
target matches the Nix host platform's minimum macOS version. The Rust toolchain
does not propagate Nix's replacement system libraries into the link environment.
Gateway and supervisor binaries staged into branch E2E, Release Dev, and Release
Tag images are compiled through `cargo auditable` (pinned in `mise.toml`), which
embeds a `.dep-v0` section describing the Rust dependencies actually compiled
@@ -189,15 +138,14 @@ is a different artifact from the source SBOM produced by `syft dir:.` in
`tasks/sbom.toml`, which describes the checkout, and from the image SBOM
attestation below, which describes a published image.
The shared binary build action uses the default Nix shell and compiles release
artifacts with `cargo auditable build --target <triple>`. Verification and upload
read binaries from `target/<triple>/release/`.
The shared binary build action compiles release artifacts with `cargo auditable`.
Branch E2E, Release Dev, and Release Tag image jobs stage those same artifacts
instead of rebuilding binaries in Docker. Each binary build scans its output with
Syft and requires at least one decoded Cargo package before uploading the
artifact. The action checks each binary's `--version` output and leaves its
linkage as produced by the Nix toolchain, without post-link rewriting or
platform-specific linkage checks. The CI image gains the pinned `cargo-auditable`
artifact. Darwin builds replace Nix's `libiconv` load command with the macOS
system install name, ad-hoc sign the modified binary, and fail if `otool -L`
reports any remaining `/nix/store` dependency. Runtime and Syft verification
run after that normalization. The CI image gains the pinned `cargo-auditable`
tool through `mise install --locked` but ships no auditable OpenShell binary of
its own.
@@ -221,6 +169,8 @@ Runtime layout:
gateway bundles z3, so the image does not need a distro-provided z3 runtime.
The base is pinned to a multi-architecture digest; distro security updates
require refreshing that digest and rebuilding the gateway image.
Updating the container's glibc package does not raise the binary's glibc
compatibility floor.
- **VM driver**: host GNU-linked binary installed at
`/usr/libexec/openshell/openshell-driver-vm` in Linux packages and published
as a release artifact. Linux GNU VM driver binaries must not reference
@@ -231,16 +181,12 @@ Runtime layout:
cache action runs. An explicitly configured VM runtime bundle is required to
contain every non-empty embedding input; the driver build fails before
packaging when an input is absent or empty.
- **Supervisor**: Alpine base with `nftables`; base packages are upgraded before
installing firewall tools to pick up distro security fixes. Static binary at
`/openshell-sandbox` (musl by default; see `SUPERVISOR_LIBC` above). Static
linkage keeps the binary usable when the image is mounted/extracted into
sandbox environments (Docker extraction, Podman image volumes, Kubernetes
init-container copy-self), whose libc and glibc version are not known at build
time, while `nftables` supports Kubernetes supervisor sidecar egress
enforcement. The VM driver bundles its own supervisor build
(`tasks/scripts/vm/build-supervisor-bundle.sh`) and does not read
`SUPERVISOR_LIBC`.
- **Sandbox**: Alpine-based `openshell/sandbox` image containing the static
musl `/openshell-sandbox` binary and its static VM guest-init helper.
Drivers stage this binary into the workload trust domain.
- **Supervisor**: Debian-based `openshell/supervisor` image containing only the
dynamically linked GNU `/openshell-supervisor` binary. GNU supervisor builds
must not reference `GLIBC_*` symbols newer than `GLIBC_2.28`.
Gateway image builds bake the corresponding supervisor image tag into the
gateway binary so Docker sandboxes do not depend on `:latest` by default.
@@ -281,6 +227,8 @@ The Nix test guest harness under `nix/test-guest` boots native-architecture clou
through QEMU for package, release, and E2E validation. A prepared cache entry is
captured after the exact ordered Ansible configuration list and before
test-specific packages, copied binaries, forwarded ports, or commands.
On macOS, the test guest and tmachine paths use the same pinned QEMU and OVMF
package set so the hypervisor and firmware remain compatible.
Prepared disks are flattened, sanitized QCOW2 images. The local cache keeps them
read-only and each test receives a fresh writable overlay and cloud-init
@@ -296,6 +244,23 @@ CLI conformance runs after target provisioning and operates only through the
configured OpenShell CLI. The smoke scenario verifies the black-box sandbox
lifecycle by creating, inspecting, executing in, and deleting a sandbox.
The `tests/tmachine` setup and installation caches include a digest of the
entire directory containing `ANSIBLE_CONFIG`, including local roles, task
includes, templates, inventory, and requirements. The digest uses sorted
relative paths, file contents, and executable permissions; source symlinks
are unsupported. Both keys also retain the ordered playbook paths and contents,
their base disk contents, and whether Galaxy is enabled; installation keys
include named binary inputs. The top-level `.roles` directory is excluded:
Galaxy release pins in `requirements.yaml` are treated as immutable, including
any transitive dependency pins. Cache misses with Galaxy enabled reinstall
the required roles and their dependencies before running playbooks.
The `tests/artifacts.nix` helpers build the CLI, conformance CLI, and sandbox
with musl, and the gateway and supervisor with GNU. Image assembly stages
the gateway, sandbox, and supervisor as separate binaries for their respective
Dockerfiles. The Ubuntu Docker and Fedora Podman scenarios import both local
runtime images and configure the gateway to use them.
## Python Wheel Packaging
The generated protobuf/gRPC stubs under `python/openshell/_proto/` are gitignored
@@ -367,32 +332,12 @@ Triggers differ by workflow: `.github/workflows/workflow-security.yml` runs on
`.github/workflows/codeql.yml` runs nightly on the default branch (`main`) via
`schedule`, with `workflow_dispatch` kept for manual diagnostics; and
`.github/workflows/codex-security.yml` runs on pushed `v*.*.*-pre.*` tags, and is
also callable through `workflow_call` and `workflow_dispatch`. CodeQL has no
automatic `pull_request`, `merge_group`, or push trigger, so it reports
also callable through `workflow_call` and `workflow_dispatch`. CodeQL does
not run on `pull_request`, `merge_group`, or pushes to `main`, so it reports
repository-level Code Scanning state on the default branch instead of per-PR
results, and its four-language matrix stays off the per-change critical path.
Codex Security is release-scoped rather than change-scoped, so it never runs on
a pull request or merge group automatically.
`.github/workflows/security-scan.yml` is a manual and reusable parent for Codex
Security, CodeQL, Trivy, Cargo Deny, and Actionlint/Zizmor. Each child runs
independently against the supplied pre-release tag; Codex retains its cumulative
stable-to-candidate range. The parent forwards OCI references to Trivy and
publishes SARIF, including Codex results on manual parent runs. Codex publishes
against the candidate commit on `main`; the other SARIF producers use the
candidate tag and commit. Cargo Deny retains its self-hosted runner and CI
container. Dependency Review and Trivy Changes remain separate comparison
workflows. Existing standalone triggers remain active.
The parent enforces HIGH/CRITICAL findings for Codex, CodeQL, Trivy, and Zizmor.
Its `allow-high-critical` input disables that threshold while keeping execution
and publication failures fatal. Each scanner evaluates its existing report
after publication. Actionlint stays informational; Cargo Deny runs its native
advisory check, which is independent of the severity override. Standalone
finding policies are unchanged. Child concurrency groups distinguish the
scanner as well as the caller, so sibling workflows cannot cancel one another.
Codex retains its repository-wide release qualification group.
See [CI.md](../CI.md#run-the-security-scans-together) for invocation examples.
a pull request or merge group.
- **Actionlint and Zizmor** analyze the workflow definitions themselves.
Repository configuration lives in `.github/actionlint.yml` (self-hosted runner
@@ -415,8 +360,8 @@ See [CI.md](../CI.md#run-the-security-scans-together) for invocation examples.
integration targets the cfg override does not reach. Examples remain in scope,
and E2E test code stays excluded because `e2e/` is not an analyzed path. Only
Go requires a build; the other languages use build mode `none`. Analysis runs
on the nightly schedule, by manual dispatch, or through `workflow_call`.
Results are uploaded to Code Scanning and always retained as workflow artifacts.
on the nightly schedule or by manual dispatch. Results are uploaded to Code
Scanning and always retained as workflow artifacts.
- **Codex Security** qualifies release candidates rather than individual
changes. The job installs a pinned `@openai/codex-security` release into the
runner temp directory before the repository is checked out and invokes it by
@@ -482,9 +427,7 @@ See [CI.md](../CI.md#run-the-security-scans-together) for invocation examples.
`CODEX_SECURITY_STATE_DIR`, where every shell command the agent ran is
recorded. No command at all is the signal that the sandbox failed to start.
Findings do not fail standalone informational runs; reusable callers can enable
HIGH/CRITICAL enforcement with `fail-on-findings`. Scanner and build failures
always fail. A scanner that
Findings never fail these checks; scanner and build failures do. A scanner that
cannot run, a CodeQL analyzer that does not complete, an unexpected Dependency
Graph API error, and a Codex Security range, scan, or export failure are all
errors, which keeps an informational check from silently degrading into a no-op.
@@ -496,53 +439,33 @@ job republishes the analysis job's outcome as the
required statuses, so they do not gate merges.
Codex Security findings are informational during the observation phase, and the
workflow only reports on candidates that already exist. Creating pre-release
tags and gating stable promotion on qualification results are part of
[RFC 0014](../rfc/0014-release-stability/release-qualification.md) and are not
implemented yet.
workflow only reports on candidates that already exist. Gating stable promotion
on qualification results remains proposed in
[RFC 0014](../rfc/0014-release-stability/release-qualification.md).
## Artifact Scanning
`release-auto-tag.yml` runs at 14:00 Europe/Zurich on weekdays (including daylight
saving time changes) and supports manual dispatch. Maintainers start weekday
pre-release publishing by tagging the initial `vX.Y.Z-pre.1` release candidate.
The workflow increments the highest release series' pre-release number on `main`
only when that seed exists, its stable tag does not exist, and new commits are
available. It never chooses a minor or patch version or creates the initial seed.
After pushing the tag, it explicitly dispatches `release-tag.yml` to build the
candidate.
Two entry points share `tasks/scripts/trivy-scan.sh`: a standalone analysis
workflow and a pull-request change gate. Nix supplies Trivy, Helm and `yq`.
The standalone workflow scans deployment configuration and supplied OCI
references independently of release publication. Detailed JSON reports feed the
differential gate; the summary and published SARIF consolidate configuration
findings across profiles while preserving resource identity and affected profiles.
Images and packaged charts retain separate identities based on their full
references. Publication batches respect GitHub's limit of 20 SARIF runs.
The PR/merge-group gate scans base and candidate with the same scanner and rejects
new `HIGH` or `CRITICAL` configuration findings. Its stable
`OpenShell / Trivy Changes` status succeeds when nothing relevant changed.
For PRs, the baseline is the first parent of the exact merge commit being tested,
not the event's potentially older base SHA or the current branch tip. Change
detection and both scans use that same immutable pair, including on reruns.
Merge groups and manual runs retain their explicit baseline and always scan.
Image CVEs need the standalone scan. The reporting and gate invariants are:
- A structurally invalid Trivy report is an error, not an empty finding set.
- Scanner failures prevent publication of incomplete analyses; findings alone
do not prevent publishing complete reports.
- Findings compare per profile against the same baseline profile, by semantic
identity and count rather than line number; a profile absent from the baseline
falls back to that identity's maximum across all profiles.
- The candidate's ignore file is validated, but the baseline's policy applies to
both scans, so an exemption takes effect only after merge.
See [CI.md](../CI.md#artifact-scanning) for profiles, report paths, severity
settings, exceptions, and the contributor and maintainer workflows.
See `CI.md` for the contributor workflow, labels, and maintainer merge-queue workflow.
## Docs Site
Published docs live in `docs/`, and Fern site configuration lives in `fern/`. See [fern/README.md](../fern/README.md) for the source layout, local development commands, version model, and publishing workflows.
Published docs live in `docs/`. Navigation lives in `docs/index.yml`. Fern site
configuration, components, theme assets, and publish settings live in `fern/`.
Use `mise run docs` for strict validation and `mise run docs:serve` for local
preview. PR previews are produced by `.github/workflows/branch-docs.yml` when
Fern credentials are available. Production docs publish from the release tag
workflow.
## Validation Expectations
Rust CI rejects stale or modified Cargo lockfiles and runs Clippy for the
workspace, E2E crate, and standalone examples.
- Run `mise run pre-commit` before committing.
- Run `mise run test` after code changes.
- Run `mise run e2e` for sandbox, policy, driver, or deployment changes when the
architecture/compute-runtimes.md
+92 -172
View File
@@ -1,10 +1,18 @@
# Compute Runtimes
Compute runtimes create, stop, start, delete, and watch sandbox workloads for the
gateway. Supervisor-controlled runtimes start a workload that runs the
`openshell-sandbox` supervisor, which enforces the sandbox contract locally.
Driver-controlled runtimes apply the canonical sandbox policy while
provisioning and report workload readiness directly.
gateway. A supported runtime provisions `openshell-sandbox` inside the workload,
`openshell-supervisor` outside it, a protected channel between them, and an
independent outer network fence. Drivers do not implement policy evaluation.
Podman provisions a paired workload and supervisor container using its native
libpod API. The workload uses `network=none`; the external supervisor alone joins
the configured network. A per-sandbox named volume carries their mutually
authenticated gRPC Unix socket, with supervisor credentials kept in its separate
filesystem. Both containers run as the resolved non-root identity with all
capabilities dropped. They share only a user namespace for volume ownership,
not PID, mount, or network namespaces. Podman owns paired lifecycle and health;
the common protocol owns process, identity, TCP, DNS, and forwarding semantics.
## Driver Contract
@@ -12,11 +20,15 @@ Each runtime receives a sandbox spec and canonical policy from the gateway and
is responsible for:
- Selecting the sandbox image.
- For supervisor-controlled runtimes, injecting sandbox identity and gateway
callback configuration, supplying callback credentials, and providing the
supervisor binary or image.
- For runtimes without the standard supervisor, validating and applying the
canonical policy before launching the workload.
- Resolving an immutable non-root sandbox identity before workload creation.
- Supplying separate sandbox and supervisor bootstrap material.
- Delivering `openshell-sandbox` to the workload and `openshell-supervisor` only
to the external supervisor placement.
- Provisioning protected control and boundary configs plus a private Unix socket,
TLS-authenticated TCP, or vsock transport when the supervisor is separated.
Runtime-specific code supplies immutable resource claims and transport
coordinates; the shared boundary protocol supplies lifecycle, exec, signaling,
forwarding, and binary identity semantics.
- Forwarding the exact canonical main-process argv and TTY mode without shell
reconstruction. The sandbox-level environment and policy workspace apply to
the main process.
@@ -127,32 +139,19 @@ defines the available implementation set, while the runtime consumes a generic
registry. Adding or removing a compiled driver therefore changes registration
rather than the server's selection flow. Alternate gateway binaries can install
their own `ComputeDriverFactory` registrations and hand the completed registry
to `run_cli_with_compute_drivers`. Factories expose the same side-effect-free
configuration validation to package preflight and runtime startup, receive only
the selected `[openshell.drivers.<name>]` table, and return either an in-process
driver or a gateway-managed remote endpoint when built. Existing out-of-tree
factories that implement only runtime construction remain source-compatible and
can start normally, but package preflight fails closed until they explicitly
advertise and implement side-effect-free configuration validation. If a selected config file
omits the selector, preflight validates each configured table that belongs to an
auto-detectable registration. It does not run runtime detection probes because
those probes can connect local sockets or launch bounded discovery commands.
Preflight never builds a driver or connects to its transport. The server constructs the common runtime
adapter and snapshots `GetCapabilities` for either result. A configured UDS
endpoint still takes precedence over a compiled registration with the same name.
to `run_cli_with_compute_drivers`; factories receive merged driver config and
return either an in-process driver or a gateway-managed remote endpoint. The
server constructs the common runtime adapter and snapshots `GetCapabilities`
for either result. A configured UDS endpoint still takes precedence over a
compiled registration with the same name.
The `openshell-gateway` composition crate exposes one feature per first-party
registration: `compute-driver-kubernetes`, `compute-driver-docker`,
`compute-driver-podman`, `compute-driver-vm`, and `compute-driver-mxc`. Builds
can enable any subset. MXC links only on Windows; the other four features
install rejection stubs on Windows and link their drivers on other platforms.
The default `in-tree-compute-drivers` feature remains an alias for all five,
preserving each platform's default registrations.
`openshell-server` has no compute driver dependencies or backend-name dispatch.
Protocol-only gateway builds disable the default features and link no
compute-driver crates. E2E lanes compose that gateway with Docker, Podman,
Kubernetes, and VM driver executables over the public UDS gRPC contract so an
in-tree driver cannot silently depend on a server-only API.
The `openshell-gateway` composition crate groups first-party registrations
behind the `in-tree-compute-drivers` feature. `openshell-server` has no compute
driver dependencies or backend-name dispatch. Protocol-only gateway builds
disable the composition feature and link no compute-driver crates. E2E lanes
compose that gateway with Docker, Podman, Kubernetes, and VM driver executables
over the public UDS gRPC contract so an in-tree driver cannot silently depend
on a server-only API.
## Stop and Start Lifecycle
@@ -177,6 +176,12 @@ same resource. The gateway requires a fresh supervisor session before a
starting sandbox returns to `Ready`; stale driver snapshots and supervisor
sessions cannot promote a `Stopped` row.
Runtime credentials are generation-scoped and memory-only after launch. A
supervisor or Sandbox Runtime process replacement does not resume a running
generation. Planned upgrades stop the sandbox first; the following start mints
a fresh session, TLS identity, and credential pair. An unexpected replacement
leaves the old workload on the normal fail-closed disconnect path.
A driver stop operation does not complete while its backend still reports an
in-progress stop. This prevents an immediate start from racing the previous
run's delayed exit event and regressing the new run to `Error`.
@@ -260,11 +265,11 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
| Runtime | Best fit | Sandbox boundary | Notes |
|---|---|---|---|
| Docker | Local development with Docker available. | Container plus nested sandbox namespace. | Uses host networking so loopback gateway endpoints work from the supervisor. Advertises the combined-supervisor policy-DNS and transparent-TCP substrate. |
| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API and CDI GPU devices when available. Delivers the supervisor via OCI image volume by default; falls back to extracting the binary to a host-side cache and bind-mounting it when `userns` is configured (overlay does not support idmapped mounts). Advertises the combined-supervisor policy-DNS and transparent-TCP substrate. |
| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
| VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a custom `compute_driver = "<name>"` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--compute-driver <name>` with `--compute-driver-socket=<path>`. A launch-time endpoint may use a canonical built-in name to preserve its driver-config key while replacing in-process construction. The gateway connects to an operator-provisioned UDS, snapshots `GetCapabilities`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
| Docker | Local development with Docker available. | Capability-free workload container. | Uses `network_mode=none`; a separate capability-free supervisor container mediates egress and access over a private daemon-local Unix socket volume. |
| Podman | Existing rootless driver. | Container. | Not converted by this isolation stack. |
| Kubernetes | Cluster deployment through Helm. | Capability-free sandbox Pod. | Uses one namespace-wide empty-egress workload NetworkPolicy and a separate capability-free supervisor Pod over mutually authenticated TLS. It requires an enforcing CNI and trusted sandbox namespace. |
| VM | Experimental microVM isolation. | Per-sandbox libkrun or QEMU VM. | The NIC-less guest runs `openshell-sandbox` as PID 1; host `openshell-supervisor` owns gateway networking and reaches the guest over vsock. |
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. A launch-time endpoint may use a canonical built-in name to preserve its driver-config key while replacing in-process construction. The gateway connects to an operator-provisioned UDS, snapshots `GetCapabilities`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
Per-sandbox CPU and memory values currently enter the driver layer through
template resource limits. Docker and Podman apply them as runtime limits.
@@ -291,42 +296,17 @@ operator override because they place gateway-host filesystem state inside the
sandbox and can negate OpenShell workspace isolation and filesystem-policy
controls. Driver-owned supervisor, token, and TLS bind mounts stay reserved.
Network features follow the existing driver/substrate split. Compute drivers
advertise only the runtime mechanics they can guarantee: namespace and
capability ownership, DNS/TCP capture installation, and coupled
restart ordering. The shared supervisor remains the sole owner of DNS
eligibility, synthetic mappings, process authorization, destination filtering,
pinned dialing, relay behavior, and OCSF decisions. Docker and Podman advertise
`policy-dns-transparent-tcp`; other runtimes reject explicit TCP policy until
they implement and validate the same complete contract. The capability marker
is driver-owned supervisor input and is removed from workload environments.
Kubernetes, Docker, and Podman share one AppArmor configuration model:
`RuntimeDefault`, `Unconfined`, or `Localhost/<profile>`. Each driver translates
that model to its native API and rejects an explicitly requested confined
profile when its backend reports AppArmor unavailable. Docker keeps its
historical explicit `Unconfined` default. Podman sends no override when the
field is omitted, preserving the runtime-selected profile; development paths
that require the supervisor's namespace mount setup opt into `Unconfined`
explicitly. The Helm chart independently uses `Unconfined` for Kubernetes.
Corporate proxy settings are driver-owned supervisor inputs. Docker, Podman,
and VM propagate `https_proxy`, `no_proxy`, an optional root-only auth file,
and the explicit cleartext-Basic-auth acknowledgement without allowing
workload environment to override them. Podman and VM can also project an
operator CA bundle for an HTTPS or TLS-intercepting proxy. The VM driver validates and
stages its credential and CA bundle under fixed guest paths, then forwards
those paths through the protected supervisor argument file rather than the
guest environment. Local containers project provider SPIFFE
through a dedicated host UNIX-socket parent mount. A VM cannot safely expose
that host socket: it accepts only a separately operated, concrete TCP listener
when `provider_spiffe_allow_guest_tcp = true` explicitly acknowledges guest
access. Host-only sockets are never implicitly forwarded to VM guests.
Network features follow the driver/substrate split. Drivers own only the outer
fence and protected channel. The sandbox owns seccomp notification, local DNS,
socket virtualization, process observation, and binary identity. The supervisor
owns DNS eligibility, policy authorization, destination filtering, upstream
dials, relay behavior, credential rewriting, and OCSF decisions. No supported
path requires nftables, a workload network namespace, proxy environment
variables, added capabilities, or an unconfined AppArmor profile.
The Kubernetes deployment packaging has two ownership boundaries. The gateway
chart owns the gateway workload, configuration, Services, PKI, and
cluster-scoped gateway resources. It can retain the legacy combined behavior,
or omit workspace resources. The workspace chart is installed into a
cluster-scoped gateway resources. The workspace chart is installed into a
pre-provisioned sandbox namespace and owns only the sandbox ServiceAccount,
namespaced RBAC, and sandbox ingress NetworkPolicy. Its RoleBinding names the
gateway ServiceAccount and namespace explicitly, so the two releases have
@@ -339,14 +319,10 @@ can request a specific number of GPUs or the driver-specific default behaviour.
For all in-tree drivers, this is equivalent to selecting a single GPU.
VM runtime state paths are derived only from driver-validated sandbox IDs
matching `[A-Za-z0-9._-]{1,128}`. Each writable overlay records its effective
sandbox UID/GID so later rootfs cache changes cannot rewrite persisted file
ownership. Unmarked pre-migration overlays recover identity from concrete
overlay or prepared-rootfs state, an explicit operator override, or the current
image account. The driver never assumes `10001:10001`; it preserves that legacy
identity only when persisted state reports it. The gateway-owned VM driver
socket uses a private `run/` directory plus Unix peer UID/PID checks. Standalone unauthenticated TCP
mode is disabled unless explicitly enabled for local development.
matching `[A-Za-z0-9._-]{1,128}`. The gateway-owned VM driver socket uses a
private `run/` directory plus Unix peer UID/PID checks. Standalone
unauthenticated TCP mode is disabled unless explicitly enabled for local
development.
Runtime-specific implementation notes belong in the driver crate README:
@@ -355,113 +331,57 @@ Runtime-specific implementation notes belong in the driver crate README:
- `crates/openshell-driver-kubernetes/README.md`
- `crates/openshell-driver-vm/README.md`
The combined VM topology runs `openshell-sandbox` as guest PID 1. libkrun
executes the driver-owned guest bootstrap as PID 1, and the bootstrap preserves
that identity when it execs the supervisor after mounting and network setup.
The VM guest bootstrap runs once as root to prepare mounts, loopback, and the
safe port-53 sysctl. It then drops to the resolved identity with empty
capability sets and executes `openshell-sandbox` as guest PID 1.
## Supervisor Delivery
The supervisor must be available inside each sandbox workload:
Drivers deliver the two binaries to separate trust domains:
| Runtime | Delivery model |
|---|---|
| Docker | Bind-mounted local supervisor binary, or a binary extracted from the configured supervisor image. |
| Podman | Read-only OCI image volume by default; host-cached bind mount when `userns` is configured. |
| Kubernetes | Supervisor image side-loaded into the sandbox pod by image volume or init container. |
| VM | Embedded in the guest rootfs bundle. |
| Docker | A digest-pinned daemon-local volume supplies `openshell-sandbox`; the companion image runs `openshell-supervisor`. |
| Podman | Existing driver behavior; not converted by this stack. |
| Kubernetes | A non-root init container stages `openshell-sandbox` into a memory volume; a directly managed Pod runs `openshell-supervisor`. |
| VM | `openshell-sandbox` is embedded in the guest rootfs; a separately digest-checked native `openshell-supervisor` runs on the host. |
| Extension | Defined by the out-of-tree driver. |
Driver-controlled environment variables must override sandbox image or template
values for sandbox ID, sandbox name, gateway endpoint, relay socket path, TLS
paths, and command metadata.
Driver-controlled sandbox bootstrap must override image or template values for
sandbox identity, command metadata, resolver configuration, and public trust
paths. Gateway endpoints, callback credentials, policy, and private TLS material
belong only to the supervisor placement.
## Process Identity
The gateway preserves whether each policy process field was omitted. The active
driver then supplies one authoritative identity input to the supervisor:
The gateway preserves whether each policy process field was omitted and passes
the admitted selectors to the driver. The driver resolves one exact UID, GID,
and supplementary-group set before creating the immutable workload:
- Docker and Podman inspect the final sandbox image, pin container creation to
its immutable image ID, and pass its raw OCI `Config.User`. Docker also
resolves the workspace from OCI `Config.WorkingDir` during that inspection.
- Kubernetes passes its platform-resolved numeric UID/GID, including OpenShift
SCC-derived values.
- VM keeps its existing guest identity behavior.
- Docker pins the image ID, resolves policy selectors against the image's
`/etc/passwd` and `/etc/group`, and validates its OCI working directory.
- Kubernetes uses platform-resolved numeric values, including OpenShift
namespace ranges.
- VM uses the configured numeric guest identity.
Explicit numeric workload identities may use any Linux UID/GID from `1`
through `u32::MAX - 1`. UID/GID `0` remains prohibited as root, and
`u32::MAX` remains prohibited because Linux APIs and POSIX ACLs use it as an
invalid identity sentinel. Infrastructure identities use separate validation:
the Kubernetes network proxy UID remains at least `1000` and must not match the
workload UID because its traffic bypasses the pod egress fence.
UID/GID zero and `u32::MAX` are invalid. The sandbox and every child start with
the resolved identity and zero capability masks; neither process performs an
in-workload UID transition. Identity-changing policy updates require sandbox
recreation, while other policy updates remain live.
For Docker and Podman, policy values take precedence independently. An omitted
`run_as_user` or `run_as_group` falls back to the corresponding identity from
the image. The supervisor resolves names from the image's `/etc/passwd` and
`/etc/group` before readiness, preserves declared name or numeric components,
and uses the same privilege-drop path for direct and SSH children. When a
declaration omits the group, the supervisor fills it with the user's numeric
primary GID. It does not rewrite the account files.
Docker uses an absolute OCI working directory as the workspace. Empty, root,
and explicit `/sandbox` values select `/sandbox`; other paths must already
exist without symlink or reserved-mount collisions and must be usable by the
resolved identity. Kubernetes and VM use `/sandbox`.
Docker uses an absolute OCI working directory as the workspace. An
empty, root (`/`), or explicit `/sandbox` declaration uses `/sandbox`, which
OpenShell creates and owns as a compatibility workspace. Any other workdir must already
exist in the immutable image without symlink components. The completed
identity, including supplementary groups, must already be able to traverse
every parent and write and enter the workdir; OpenShell does not change that
directory's ownership or mode. A one-shot validator drops to that identity and
uses kernel effective-access checks so POSIX ACL and LSM decisions are honored.
Path checks reserve the standard OCI runtime namespaces under `/proc`, `/sys`,
and `/dev`, while separate collision checks are derived from actual OpenShell
control paths.
Docker performs the check in the final container before workload launch and
rejects image `VOLUME` declarations that would mask the workdir ancestry. The
resolved workspace is the child cwd and `HOME`; when
`filesystem.include_workdir` is enabled, it becomes the automatic writable
policy path. Podman, Kubernetes/OpenShift, and VM retain their existing
`/sandbox` workspace behavior.
Sandbox creation fails before the workload becomes ready when a required image
identity is absent, malformed, unknown, ambiguous, or resolves to UID/GID 0.
The supervisor itself remains root so it can establish isolation before
starting unprivileged children.
Kubernetes can run the supervisor in the default combined topology or in a
sidecar topology. Combined mode keeps network and process supervision in the
agent container. Sidecar mode runs network enforcement, the proxy, and gateway
session in a dedicated sidecar, while the agent container runs only the
process-supervision leaf and launches the user workload after the sidecar
serves bootstrap state over a local control socket. The network sidecar owns
gateway credentials and sends policy plus workload-facing provider environment
state to the process leaf over that socket. It also streams provider
environment updates after settings polls so future process sessions see
updated provider env without giving the process leaf gateway access. The
pre-workload process supervisor is the only accepted control client: the
network sidecar verifies its UID, GID, and PID with peer credentials, removes
the listener after accepting it, and ignores workload-supplied relay targets.
SSH relays use a Linux abstract socket and verify its peer PID against that
authenticated process-supervisor connection, so workload filesystem access
cannot replace the relay endpoint. Either supervisor exits when this control
connection closes. This couples their restart lifecycle and prevents a workload
that survives an isolated network-sidecar restart from becoming the next
authoritative control client. In sidecar mode, an init container performs the
privileged pod-network nftables setup with
`NET_ADMIN`. The default binary-aware network sidecar runs as UID 0 without
`NET_ADMIN` and adds `SYS_PTRACE` plus `DAC_READ_SEARCH` so it can resolve
cross-UID workload process/binary identity through shared `/proc`. Operators
can set the sidecar `process_binary_aware_network_policy` flag false to run the
sidecar as the configured non-root proxy UID, omit both inspection capabilities,
and downgrade network policy to endpoint/L7 matching without `policy.binaries`.
The init path applies nftables as individual commands so optional conntrack and
log expressions can fail without rolling back the required table, chain, and
reject rules.
The agent container runs as the resolved sandbox UID/GID with no added Linux
capabilities. Sidecar mode preserves gateway session and SSH behavior, but
treats the process leaf as network-only: Landlock filesystem policy and child
seccomp still apply where supported, while process privilege dropping and
supervisor identity mount isolation do not run because the agent container is
already unprivileged. Sidecar pods use a shared process namespace so the
network sidecar can resolve workload process and binary identity through
`/proc/<entrypoint-pid>`.
The Kubernetes driver creates the namespace-wide empty-egress workload fence
before a suspended Sandbox CR, then provisions split immutable bootstrap
Secrets, the private runtime Service, and a gated supervisor Pod. A
non-root init container stages `openshell-sandbox` and one-use bootstrap files
into memory volumes. The workload Pod never mounts supervisor or gateway
credentials. The driver removes its scheduling gate only after the companions
exist; measured confirmation and supervisor-session registration gate public
readiness.
## Images
architecture/gateway.md
+24 -4
View File
@@ -53,6 +53,15 @@ The gateway validates this requirement before constructing the selected driver.
## Protocol and Auth
Gateway validation and concurrency errors use the standard rich gRPC error
envelope. Shared field validators attach `google.rpc.BadRequest`, and conditional
write conflicts attach `google.rpc.ErrorInfo` with a stable reason and current
version when available. `google.rpc.RetryInfo` expresses a minimum retry delay;
it does not establish that a mutation is safe to repeat. SDKs retain the original
transport status, metadata, and unknown details alongside decoded fields.
Python cleanup inspects the original gRPC call beneath a typed error wrapper,
preserving missing-resource handling without suppressing other failures.
The gateway listens on one service port and multiplexes gRPC and HTTP traffic.
The default local single-user deployment mode is mTLS user authentication:
clients present a certificate signed by the local deployment CA, and the
@@ -215,7 +224,7 @@ Supported auth modes:
| Plaintext | Local development or a trusted reverse proxy boundary. |
| Unauthenticated local users | Trusted Kubernetes dev or fully trusted proxy deployments only. |
| Cloudflare JWT | Edge-authenticated deployments where Cloudflare Access supplies identity. |
| OIDC | Bearer-token auth for users, with browser or device-code PKCE and client credentials login. JWKS validation accepts RS256, RS384, RS512, PS256, PS384, PS512, ES256, ES384, and EdDSA (Ed25519) signing keys. |
| OIDC | Bearer-token auth for users, with browser or device-code PKCE and client credentials login. Discovery and JWKS retrieval require HTTPS, reject redirects, and pin JWKS to the issuer origin or an explicit origin allowlist. JWKS validation accepts RS256, RS384, RS512, PS256, PS384, PS512, ES256, ES384, and EdDSA (Ed25519) signing keys. |
The CLI persists the scopes requested during OIDC login in gateway metadata and
reuses them when refreshing an access token. This preserves the intended API
@@ -267,8 +276,13 @@ controlling Sandbox CR. The bootstrap path accepts
both `agents.x-k8s.io/v1beta1` ownerReferences from newer Agent Sandbox
controllers and `agents.x-k8s.io/v1alpha1` ownerReferences from existing
deployments. Supervisors renew gateway JWTs in memory before expiry only while
the sandbox record still exists. Older tokens are not server-revoked; shared
deployments bound replay exposure with short `gateway_jwt.ttl_secs` lifetimes.
the sandbox record still exists. Each successful refresh atomically stores the
new gateway-token ID in that sandbox record. The immediately consumed bearer
can recover that same successor for 30 seconds when the request matches, but it
cannot authorize ordinary RPCs or choose another successor. Advancing the
successor removes that retry path across every gateway replica. Short
`gateway_jwt.ttl_secs` lifetimes still bound the exposure of a current bearer
that has not yet been refreshed.
Omitting `gateway_jwt.ttl_secs` selects non-expiring tokens for local
single-player Docker, Podman, and VM gateways; those tokens carry `exp = 0`.
Kubernetes and other shared deployments should set a positive TTL. Explicit
@@ -464,6 +478,12 @@ credential storage for defense in depth. Multi-replica deployments can use that
default with a shared database and shared key-encryption key, or opt into an
external backend such as Vault or Kubernetes Secrets.
The Vault credential driver requires HTTPS for every non-loopback backend,
never follows HTTP redirects, and keeps standard certificate hostname
verification enabled. Operators can add private Vault trust roots with a PEM
CA bundle; the driver does not replace platform roots or expose a certificate
verification bypass.
Sandbox workload templates are workspace-scoped gateway resources. Workspace
admins create and delete them; workspace users can read and list them. A
template owns reusable workload intent: image, environment, CPU and memory
@@ -933,7 +953,7 @@ system entry instead of pretending to delete package-manager owned state.
- Compute runtimes own the mechanics of starting workloads and injecting
callback configuration. Local Docker, Podman, and VM callback endpoints can
be derived from their fixed host aliases. Kubernetes requires an explicit
endpoint from deployment topology; Helm renders it from the gateway Service
endpoint from driver placement; Helm renders it from the gateway Service
name and namespace rather than inferring it from sandbox placement.
- Docker-backed local gateways use Docker's `host-gateway` callback alias on
macOS and Docker Desktop-style runtimes. They request IPv4 loopback callback
architecture/sandbox.md
+285 -189
View File
@@ -1,43 +1,95 @@
# Sandbox
A sandbox is the runtime boundary where agent code executes. It is created by a
compute runtime and managed inside the workload by `openshell-sandbox`, the
sandbox supervisor.
A sandbox is the runtime boundary where agent code executes. A compute driver
creates it and connects two dedicated components: `openshell-sandbox` inside
the workload boundary and `openshell-supervisor` outside it.
## Runtime Model
Each sandbox workload has two trust levels:
Each sandbox has three trust levels:
| Process | Role |
| Component | Role |
|---|---|
| Supervisor | Starts as root inside the workload, prepares isolation, runs the proxy, fetches config, injects credentials, serves the relay socket, and launches child processes. |
| Agent child | Runs as an unprivileged user with filesystem, process, and network restrictions applied. |
| Supervisor | Owns gateway credentials, admitted policy, L7 proxying, SSH, and gateway relays. It never executes inside the agent workload. |
| Sandbox | Runs as the same non-root identity as the agent, installs the workload seccomp listener, applies the Landlock baseline, owns child processes, and mediates the protected supervisor channel. |
| Agent child | Inherits the sandbox network listener and runs with zero capabilities, `no_new_privs`, Landlock, and the final syscall filter. |
The supervisor keeps enough privilege to manage the sandbox, but the agent child
loses that privilege before user code runs. On Linux, child setup clears the
capability bounding set during privilege drop so later execs cannot regain
container-granted capabilities. This is fail-closed: the supervisor retains
`CAP_SETPCAP` solely to perform the clear, and spawning the workload or SSH shell
aborts unless the bounding set ends up empty. A `setpcap` `EPERM` is tolerated
only when the set is already empty; any other outcome fails the spawn.
The runtime grants neither trusted component nor agent child any Linux
capability inside the workload. Drivers resolve one exact non-root UID, GID,
and supplementary-group set before launch. The sandbox and all of its children
use that immutable identity, so no in-workload privilege transition is needed.
The supervisor uses its own driver-defined identity and has no workload-creation
or backend-admin authority.
The compute driver provisions separate protected configurations and one
mutually authenticated gRPC connection over a private Unix socket, Kubernetes
TCP Service, or VM vsock channel. Independent bidirectional `Exchange` RPCs
carry lifecycle, exec, TCP, and forwarding traffic, while one persistent
bidirectional `Mediate` RPC carries multiplexed DNS traffic. General application
UDP is unsupported; UDP DNS remains mediated by the supervisor.
The sandbox probes HTTP/2 connection liveness every five seconds and closes
connections that miss a ten-second acknowledgement deadline. Closing a
connection freezes the owned workload process tree and cancels its stream
bridges before releasing the exclusive DNS mediation lease. The supervisor has
30 seconds to reconnect, replay attach, and reconfirm the boundary. Every
supervisor process generates an ephemeral instance ID, and the sandbox pins the
first ID it accepts for its process lifetime. The same process can therefore
recover a dropped transport, but a replacement supervisor cannot reuse launch
credentials to claim the existing runtime generation. Confirmation resumes the
workload; expiration terminates it. A credential replacement does not displace
the active connection until the new connection is confirmed. Idle healthy
connections remain usable.
Unauthenticated TLS handshakes have a separate bounded asynchronous pool and
five-second deadline, never consuming authenticated control slots or threads.
The socket broker reserves the TCP control-listener port against workload
connections, including loopback aliases. Unix control listeners reject workload
descendants using kernel peer credentials and process ancestry, while ordinary
workload loopback and Unix services remain available.
NetworkPolicy is an outer reachability fence, not a confidentiality boundary.
Each sandbox generation receives a fresh CA and distinct server/client leaves;
both endpoints bind the same workload identity and immutable driver resource
claims. Driver crates do not appear in generic process, network, SSH, or
session code.
The supervisor exposes readiness only after the sandbox is confirmed and the
gateway access plane is registered. Driver-owned channel directories limit
reachability, while mutual authentication and channel epochs prevent endpoint
replacement from granting authority.
## Startup Flow
1. The compute runtime starts the supervisor with sandbox identity, callback
endpoint, TLS or secret material, image metadata, and initial command.
2. In gateway-backed mode, the supervisor opens `ConnectSupervisor` and
receives the complete desired-state bootstrap. Standalone mode reads its
explicit local files without opening a gateway session.
3. It installs policy, settings, middleware, and provider state from that
authoritative source.
4. It prepares filesystem access, process restrictions, network namespace
routing, trust stores, and provider credential resolution.
5. It starts the policy proxy and local SSH server, then launches the persisted
canonical main-process argv and retains its PTY
or pipes in the main-session multiplexer.
6. After runtime endpoints are ready, it reports bootstrap application and
keeps the session open for desired-state delivery, connect, exec, file sync,
and log push.
1. The driver resolves the immutable workload identity, installs the outer
network fence, and starts `openshell-sandbox` with one-use bootstrap state.
2. The sandbox consumes and unlinks bootstrap material, proves the admitted
runtime posture, and listens on the protected driver channel. It does not
run untrusted code yet.
3. `openshell-supervisor` opens its gateway session and receives an
authoritative policy, settings, middleware, and provider bootstrap before
attaching to the sandbox. Compatibility protocol revisions continue to use
the polling APIs.
4. The sandbox installs its seccomp notification broker and Landlock baseline,
then reports measured confirmation. The supervisor must accept that evidence
before it sends the launch permit.
5. The sandbox starts the canonical process through its single workload
launcher. The supervisor starts SSH, acknowledges bootstrap application on
the prepared session, and exposes readiness only after gateway acceptance.
6. Later desired-state snapshots arrive on that session. Exec, signaling, PTY,
DNS, TCP, and loopback-forwarding operations cross the authenticated
boundary channel for the lifetime of the sandbox generation.
When the admitted main process exits, its status and retained terminal output
remain available. The confirmed sandbox and supervisor-owned access plane continue
to serve policy-authorized exec and loopback forwarding until explicit stop or
delete tears down the boundary and terminates any remaining workload processes.
Completed exec output handles can be reclaimed, but execution request IDs remain
reserved for the boundary generation. The sandbox accepts at most 4,096 exec
attempts per generation, then rejects new attempts rather than forgetting replay
protection. A disconnected attachment does not authorize another execution.
While an exec handle is retained, independent waits return its stable exit or
signal status, whether or not an output attachment is open or the main process
has exited. Waiting never holds the exec registry lock, so other operations can
still signal or attach to the process.
## Isolation Layers
@@ -46,10 +98,10 @@ OpenShell uses overlapping controls rather than a single sandbox primitive:
| Layer | Purpose |
|---|---|
| Filesystem policy | Landlock restricts the paths the agent can read or write. |
| Process policy | The child process runs as a non-root user with reduced privileges. |
| Seccomp | Blocks dangerous syscalls, including raw socket paths that bypass the proxy. |
| Network namespace | Forces ordinary agent egress through the local CONNECT proxy. |
| Policy proxy | Evaluates destination, binary identity, TLS/L7 rules, SSRF checks, and endpoint-bound credential injection. |
| Process policy | Sandbox and children run as one immutable non-root identity with zero capabilities. |
| Seccomp notification | Virtualizes supported INET sockets and sends DNS/TCP decisions to the supervisor without nftables or proxy environment variables. |
| Driver outer fence | Docker `network_mode=none`, a NIC-less VM, or Kubernetes NetworkPolicy prevents any missed or unsupported kernel path from escaping. |
| Policy proxy | Evaluates destination, binary identity, TLS/L7 rules, SSRF checks, and inference interception. |
The supervisor may enrich baseline filesystem allowances for runtime-required
paths, such as proxy support files or GPU device paths when a GPU is present.
@@ -68,17 +120,88 @@ invokes `wxc-exec`, rather than running a chain with missing implementations.
Remove the middleware entries or use a compute driver whose sandbox supervisor
receives the gateway middleware registry.
## Network and Provider Access
The mandatory self-protection baseline is separate from optional workload
filesystem policy. It requires Landlock ABI v3, including pathname truncation
protection. Rules cover individually opened root children except `/.openshell`;
the sandbox opens entries relative to a pinned root descriptor without following
symlinks. An image-provided alias cannot grant access to the protected subtree.
The reserved `/.openshell` root must itself be a real directory if present;
a symlink or non-directory aborts preparation so private child mounts cannot
redirect into an allowed subtree.
## Network and Inference
See [Sandbox Limits](sandbox-limits.md) for the current numeric safety ceilings,
their ownership, terminal behavior, and known gaps.
All ordinary agent egress is routed through the sandbox proxy. The proxy
identifies the calling binary, checks trust-on-first-use binary identity, rejects
unsafe internal destinations, and evaluates the active policy. On Linux, it
maps an accepted proxy connection back to the workload socket by matching the
complete local-to-remote TCP tuple before resolving every process that owns the
socket inode.
### Standalone network proxy
`openshell-supervisor --role=network-proxy` runs the policy proxy without an
Isolation Backend or `openshell-sandbox`. It accepts explicit HTTP proxy and
CONNECT requests on a loopback listener and applies the same local Rego rules,
YAML policy data, destination checks, and L7 enforcement used by supervised
sandboxes:
```shell
openshell-supervisor \
--role=network-proxy \
--listen=127.0.0.1:3128 \
--tls-dir=/tmp/openshell-proxy-tls \
--policy-rules=/path/to/sandbox-policy.rego \
--policy-data=/path/to/sandbox-policy.yaml
```
The standalone listener cannot observe which process opened a connection, so
this role evaluates endpoint and protocol rules without binary identity. It
does not launch a workload, attach a Sandbox Runtime, fetch gateway policy,
inject provider credentials, or provide exec and lifecycle operations. The
listener is loopback-only. TLS interception writes its generated public CA and
combined trust bundle to `--tls-dir`; when omitted, the supervisor uses a
process-specific directory under the system temporary directory.
The sandbox installs one seccomp user-notification listener on a dedicated
launcher thread. Every canonical and exec process inherits that listener. It
virtualizes supported INET sockets before they enter the agent FD table, copies
bounded syscall inputs from the notifying task, resolves the calling binary,
and blocks external `connect` until the supervisor returns a policy decision
and relay stream. Connected data stays on ordinary kernel sockets, so the
notification path is limited to socket setup and pointer-bearing operations.
Blocking listener accepts retain native workload socket flags. A broker-owned
watchdog interrupts an accept when its seccomp notification is cancelled or the
broker stops, including when readiness disappears before the accept syscall.
The sandbox reserves `SIGUSR2` with a non-restarting no-op handler for these
broker threads; startup rejects a conflicting handler. This signal disposition
is process-global kernel state, while registrations and cancellation state are
owned by the broker. Workload exec resets the caught handler to its default.
This sandbox runtime requires Linux 6.2 or newer for Landlock ABI v3 and treats
`SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV` as mandatory so cancelled
notifications cannot race task-memory writes.
DNS uses an exact sandbox-local resolver at `127.0.0.53:53`. The driver sets the
nameserver and permits an unprivileged bind to port 53. UDP and TCP DNS requests
are forwarded through the supervisor, which applies hostname-based DNS policy.
DNS sender identity is explicitly unavailable: native writes can come from an
inheriting process or after exec, and neither the connecting binary nor a later
descriptor-owner snapshot proves who sent an already queued query. Consumers
must not use this unavailable identity to grant binary-specific access. TCP
connection authorization still uses decision-time binary identity.
The sandbox retains only bounded DNS socket-admission records, consumes TCP
records on accept, and reclaims closed UDP records when capacity is reached.
The kernel delivers replies from the configured nameserver address, including
for strict musl and c-ares resolvers. No proxy environment variable, nftables
rule, or workload network namespace setup is part of enforcement. The supervisor
retries failed DNS accepts with backoff, preserving service across a channel
reconnect.
External TCP opens wait at most 30 seconds for a supervisor decision, then fail
with `ETIMEDOUT` and release their worker quota. An approval is tied to the
original socket identity; replacing the descriptor during policy evaluation
cannot transfer that approval to another socket.
The outer fence remains mandatory. If notification handling misses a syscall,
loses the supervisor, exceeds a bound, or encounters an unsupported socket
type, the request fails and the driver-owned fence still blocks direct egress.
CONNECT and absolute-form forward HTTP are explicit-proxy adapters over the same
egress pipeline. Each adapter normalizes its request into an egress intent, and
@@ -104,29 +227,6 @@ captured before the bypass fence, mapped back to its workload process, authorize
through the same egress pipeline, and dialed only through the pinned addresses.
Omitted protocol endpoints retain explicit-proxy behavior.
The DNS store is in-memory and sandbox-local. A combined-supervisor restart also
restarts its workload; before execution, the supervisor advances a persisted
boot epoch and installs only that epoch's synthetic capture ranges. An address
cached from the preceding epoch therefore falls through to the bypass fence
instead of inheriting a new mapping. Policy reload, expiry, wrong ports, direct real-IP access, missing
mappings, or pool exhaustion fail closed. Resolver injection, DNS listeners,
capture rules, and the transparent listener are all ready before workload
execution. A runtime that cannot provide the complete contract rejects a policy
containing explicit TCP endpoints rather than partially activating it. Because
that substrate is startup infrastructure, a sandbox created without explicit
TCP endpoints rejects a hot reload that introduces one and keeps its complete
previous policy active; recreating the sandbox installs the substrate before
the workload starts. A sandbox that started with the substrate may continue to
remove and re-add TCP endpoints through ordinary atomic policy reloads.
Workload DNS targets port 53, while nftables redirects eligible IPv4 DNS traffic
to an unprivileged supervisor listener. The filter admits DNS and transparent
TCP only when the kernel records the traffic as DNATed to the corresponding
supervisor listener, so direct dials to either unprivileged listener port remain
fenced. `SO_ORIGINAL_DST`, synthetic mapping lookup, endpoint correlation, and
generation-pinned authorization form the transparent TCP security boundary.
Docker and Podman do not currently advertise usable IPv6 egress for this
substrate, so AAAA queries return NOERROR/NODATA and IPv6 DNS remains fenced.
Provider credential placeholders are resolved through the live provider state
for each HTTP request, after destination and L7 policy admission. A static
credential resolves only when the request host, port, and path match an endpoint
@@ -139,12 +239,12 @@ partially active or last-known-good static set. Invalid metadata preserves the
supplied dynamic snapshot, while a fetch failure preserves the currently active
dynamic snapshot.
In the Kubernetes sidecar topology, the provider environment revision remains
Across the protected sandbox/supervisor channel, the provider environment revision remains
an opaque content fingerprint and has no numeric ordering semantics. The
network supervisor assigns a separate, connection-local monotonic generation
to each distinct environment it publishes. The process supervisor applies only
newer generations, which accepts descending fingerprint values while rejecting
duplicate or delayed sidecar messages.
duplicate or delayed supervisor messages.
Gateway-managed refresh credentials use an opaque workload handle derived from
the sandbox, provider identity, credential key, refresh authorization epoch,
@@ -175,7 +275,7 @@ support `params` matchers; generic JSON-RPC rules match only the method.
JSON-RPC responses and server-to-client MCP messages on response or SSE streams
are relayed but are not currently parsed for policy enforcement.
Every `protocol: mcp` endpoint carries a canonical, nonempty `mcp.versions` allowlist drawn from OpenShell's exact revision registry: `2025-03-26`, `2025-06-18`, and `2025-11-25`. A policy author may omit the entire `mcp` object when using the other endpoint defaults, or omit `mcp.versions` while setting another MCP option. Both forms resolve immediately to the exact allowlist `["2025-11-25"]`; omission never means latest or all known revisions. Defaulting applies only when the corresponding YAML key is absent: `mcp: null`, `versions: null`, and an explicit `versions: []` are invalid. At protobuf ingress, an empty repeated field means omission and uses the same default because protobuf repeated fields do not preserve presence. Normalization stores and serializes the materialized allowlist in semantic order, so adding a supported revision to the registry never widens a previously normalized policy. An explicit nonempty allowlist remains available as an advanced compatibility or downgrade control. The registry is a closed set rather than a date range, so duplicate or padded values, unknown dates, and moving aliases such as `draft` or `latest` are rejected. The sessionless `2026-07-28` revision is not accepted until OpenShell supports its distinct per-request contract. A version names a core protocol revision only; there is no policy syntax for layering a separately named SEP onto it. For every MCP HTTP request except a valid standalone `initialize`, the supervisor reads exactly one `MCP-Protocol-Version` header and requires that revision to appear in the endpoint allowlist. If the header is absent, the MCP transport specification defines `2025-03-26` as the compatibility fallback; OpenShell permits that fallback only when the allowlist contains it. Duplicate, empty, or unsupported header values receive `400 Bad Request`, while a supported revision outside the allowlist receives `403 Forbidden`. The supervisor repeats this check after middleware changes the request and before any upstream write. This check does not store session state or infer a version from a previous connection request. The registry also owns immutable batch-shape metadata: `2025-03-26` permits nonempty same-side top-level JSON-RPC batches, which OpenShell's planned enforcement caps at 64 members, while `2025-06-18` and `2025-11-25` prohibit top-level arrays. The current request parser does not yet apply these version-specific batch rules.
Every `protocol: mcp` endpoint carries a canonical, nonempty `mcp.versions` allowlist drawn from OpenShell's exact revision registry: `2025-03-26`, `2025-06-18`, and `2025-11-25`. A policy author may omit the entire `mcp` object when using the other endpoint defaults, or omit `mcp.versions` while setting another MCP option. Both forms resolve immediately to the exact allowlist `["2025-11-25"]`; omission never means latest or all known revisions. Defaulting applies only when the corresponding YAML key is absent: `mcp: null`, `versions: null`, and an explicit `versions: []` are invalid. At protobuf ingress, an empty repeated field means omission and uses the same default because protobuf repeated fields do not preserve presence. Normalization stores and serializes the materialized allowlist in semantic order, so adding a supported revision to the registry never widens a previously normalized policy. An explicit nonempty allowlist remains available as an advanced compatibility or downgrade control. The registry is a closed set rather than a date range, so duplicate or padded values, unknown dates, and moving aliases such as `draft` or `latest` are rejected. The sessionless `2026-07-28` revision is not accepted until OpenShell supports its distinct per-request runtime contract. A version names a core protocol revision only; there is no policy syntax for layering a separately named SEP onto it. The registry owns immutable batch-shape metadata: `2025-03-26` permits nonempty same-side top-level JSON-RPC batches, which OpenShell's planned enforcement caps at 64 members, while `2025-06-18` and `2025-11-25` prohibit top-level arrays. These are declared profile facts, not current forwarding claims. The allowlist does not yet select request parsing or forwarding behavior. Later response-aware runtime state must observe the successful server response, require the selected revision to be in the allowlist, and apply that one exact profile without a union or fallback; OpenShell must not bind the client proposal in `initialize` as though it were the server-selected revision.
For admitted HTTP requests, the proxy can run an ordered supervisor middleware
chain after L7 policy evaluation and before credential injection. Destination
@@ -219,9 +319,10 @@ the registry. Public custom-CA PEM travels with the stable registration.
The slots live in a supervisor-owned `ExtensionCredentialStore` shared by every
gateway connection the supervisor opens, so the registry's clients and the
configuration loop that rotates them observe the same credentials. The loop
rotates only when a credential is missing or has passed four fifths of its
lifetime, and bounds its sleep by the soonest rotation deadline.
polling loop that rotates them observe the same credentials. Configuration
polling runs far more frequently than credentials expire, so the loop rotates
only when a credential is missing or has passed four fifths of its lifetime,
and bounds its sleep by the soonest rotation deadline.
Middleware cannot observe injected credentials, introduce credential
placeholders, or mutate supervisor-owned credential, routing, or framing
@@ -233,11 +334,17 @@ security logs. See
[Supervisor Middleware](../docs/extensibility/supervisor-middleware.mdx) for
configuration and protocol details.
Inference providers use the same egress path as other external services. An
attached provider profile contributes endpoint and binary policy. The proxy
then resolves the provider's credential placeholder only when both policy and
the profile's endpoint binding authorize the native request. Model selection,
request shape, headers, streaming, and timeouts remain client concerns.
`https://inference.local` is special. It bypasses OPA network policy and is
handled by the inference interception path:
1. The proxy terminates the local TLS connection with the sandbox CA.
2. It detects known OpenAI, Anthropic, and compatible inference request shapes.
3. It strips caller-supplied credentials and disallowed headers.
4. It forwards through `openshell-router` using the route bundle fetched from
the gateway.
External inference endpoints that do not use `inference.local` are treated like
ordinary network traffic and must be allowed by policy.
In proxy-required networks, the supervisor chains upstream TLS tunnels through
a corporate forward proxy with HTTP CONNECT instead of connecting directly,
@@ -274,8 +381,9 @@ last resort for proxies whose ACLs filter on hostnames and reject IP CONNECT
targets — with it, the proxy resolves the name itself and its ACLs become
the effective egress control for proxied TLS. (Resolving through the proxy's
own DNS view, e.g. DoH tunneled via CONNECT, is a possible future
enhancement and out of scope.) The workload child's proxy variables are
unaffected — they are always rewritten to point at the local policy proxy.
enhancement and out of scope.) Workload proxy variables are removed from the
protected launch environment; transparent socket mediation does not depend on
them.
Template environment is treated like user-provided sandbox environment. It can
shape the workload child, but it cannot override driver-controlled identity,
@@ -294,10 +402,9 @@ sandbox-create time through validators shared with the supervisor
(`openshell_core::driver_utils::parse_upstream_proxy_url` and
`parse_upstream_proxy_credential`).
An optional operator CA bundle (`--upstream-proxy-ca-bundle`, a PEM path the
driver bind-mounts read-only into the sandbox) extends the trust boundary for
corporate proxies. A CA certificate is not secret, so unlike the auth file it
travels as a plain read-only bind mount rather than a driver secret. It is
An optional operator CA bundle (`--upstream-proxy-ca-bundle`, a supervisor-only
PEM path) extends the trust boundary for corporate proxies. A CA certificate is
not secret, but the supervisor is still its only configuration authority. It is
trusted in two places: the TLS handshake with an `https://` proxy, and —
because a TLS-intercepting proxy (mitmproxy, squid `ssl-bump`) re-signs
tunneled server certificates with the same CA — the sandbox combined trust
@@ -312,48 +419,28 @@ plain HTTP) and is fail-closed: an unreadable or certificate-free file is fatal.
Proxy credentials are never embedded in the URL: an inline `user:pass@` is
rejected because it would be stored in `gateway.toml` and exposed in container
metadata. Operators supply credentials via `proxy_auth_file`; the driver
stages them as a root-only secret mounted at a fixed path and passes only
stages them as a supervisor-only secret mounted at a fixed path and passes only
that path on the supervisor's command line. The supervisor reads the
file and builds the `Proxy-Authorization: Basic` header; a credential that is
empty, contains control characters, or is not in `user:pass` form is fatal on
both sides.
The VM driver has no argv seam of its own: its guest init script runs as PID 1
and execs a fixed supervisor command line, and the libkrun and QEMU launch
backends both reach the supervisor through that script. Driver-owned
supervisor arguments therefore travel in a per-sandbox file the driver writes
into the overlay upperdir at a fixed guest path, one argument per line, which
the guest reads verbatim (no word splitting or globbing) and appends to every
supervisor exec. The file is written on **every** launch, including an empty
file when there is nothing to pass: the upperdir copy always shadows the
read-only image layer, so a sandbox image can neither supply its own
supervisor arguments by baking a file at that path nor disable the operator's
by omitting one. This mirrors the driver-authored `init.d` manifest, which
solves the same trust problem for guest init drop-ins.
The VM driver starts `openshell-supervisor` on the host and
`openshell-sandbox` as capability-free guest PID 1. Corporate proxy arguments,
credentials, private CA keys, policy, and gateway credentials stay host-side.
Both libkrun and QEMU guests are NIC-less; intercepted workload connections
cross the authenticated vsock channel. A gateway-host proxy is addressed as
`host.openshell.internal`, which the host supervisor normalizes to `127.0.0.1`.
A microVM has no bind mounts or container secrets, so the VM driver stages the
credential and the CA bundle into the per-sandbox overlay disk instead — the
credential root-only, the CA world-readable, both at fixed `/opt/openshell`
paths and both removed with the sandbox state directory. The consequence,
which differs from the Podman secret model, is that the credential is at rest
inside that overlay image on the gateway host; the per-sandbox gateway JWT
already travels the same path. Proxy reachability differs by VM backend. libkrun-backed
sandboxes egress through gvproxy, so a proxy on the gateway host's loopback is
reachable through the host alias `host.openshell.internal`, which gvproxy NATs
to the host's `127.0.0.1`. QEMU/TAP sandboxes (GPU) have no equivalent: that
alias resolves to the TAP host address, and the driver's nftables `input`
chain accepts only the gateway port from the guest, so no gateway-host proxy
is reachable. The driver rejects a gateway-host proxy URL on the QEMU path at
launch rather than producing CONNECT timeouts. The guest's gateway callback is
unaffected in both backends and never traverses the proxy.
The Docker driver runs `openshell-supervisor` in a separate companion container.
Its private named volume contains supervisor bootstrap and channel material.
The workload container receives only `openshell-sandbox`, public interception
CA material, and the other sandbox half of the authenticated channel.
For Kubernetes sandboxes, the operator configures a Secret name and key rather
than a gateway-host file path. Kubernetes projects that Secret only into the
container that runs network supervision. Proxy credential Secrets require the
sidecar topology, which gives them a separate container boundary from the
workload. Combined topology is rejected because Kubernetes `fsGroup` volume
permission handling can make a shared credential mount readable by the sandbox
group.
For Kubernetes, the operator configures a Secret name and key rather than a
gateway-host file path. Kubernetes projects that Secret only into the separate
supervisor Pod. The sandbox Pod never mounts corporate-proxy credentials
or the interception CA private key.
The Basic header travels over the plain-TCP connection to the `http://` proxy,
so it is readable on the network path between sandbox host and proxy.
@@ -392,16 +479,14 @@ agent process and SSH child processes. Driver-controlled environment variables
override template values so sandbox images cannot spoof identity, callback, or
relay settings.
Supervisor bootstrap identity is not inherited by agent child processes. When
provider token grants mount a SPIFFE Workload API socket, the socket path must
live under a dedicated directory. Children also enter a private mount namespace
where that socket directory is hidden before privilege drop.
Supervisor bootstrap identity and provider workload-identity sockets never
enter the sandbox workload. The authenticated channel carries only the
policy-authorized provider environment intended for child launch and public
trust material intended for TLS clients.
Credential placeholders in proxied HTTP requests can be resolved by the proxy
when policy allows the target endpoint. For GCP providers, a loopback metadata
server inside the network namespace serves placeholders to SDKs that bypass the
proxy (e.g. Go's `cloud.google.com/go/compute/metadata`). Secrets must not be
logged in OCSF or plain tracing output. The supervisor uses revision-scoped
Credential placeholders in mediated HTTP requests can be resolved by the proxy
when policy allows the target endpoint. Secrets must not be logged in OCSF or
plain tracing output. The supervisor uses revision-scoped
placeholders for unmanaged rotating credentials and identity-stable opaque
handles for gateway-managed refresh credentials. Provider environment keys
beginning with `v<digits>_` or `s<64 lowercase hex characters>_` are reserved
@@ -494,6 +579,10 @@ Security-relevant sandbox behavior uses OCSF structured events; internal
diagnostics use ordinary tracing.
The OCSF device describes the sandbox environment, with type ID Other and type
label `Sandbox`; its operating system is a separate attribute.
HTTP Activity records contain a request or response; early rejections with only
connection context use Network Activity. Producer regression tests validate
required fields and `at_least_one` constraints against the vendored OCSF 1.8
schemas.
## Policy Proposals
@@ -512,22 +601,15 @@ quickly.
## Supervisor Configuration Delivery
The current gateway and supervisor use internal supervisor protocol revision 2.
The gateway accepts Stage 1 revision 1 supervisors through the polling
compatibility path. Peers built before the handshake report revision zero and
remain accepted for one release with a warning and counter because sandboxes
keep their supervisor binary until they are recreated. The gateway includes a
configuration bootstrap when it accepts a `ConnectSupervisor` session and can
send complete component replacements on the same stream after policy, settings,
or provider state changes.
Revision 2 supervisors require a complete bootstrap. The gateway uses the same
bounded 45-second construction window as other snapshot builds and rejects the
connection when construction fails. Revision 1 compatibility sessions retain
the optional one-second bootstrap budget and use polling when it expires.
The revision 2 supervisor opens the stream and consumes the bootstrap before it
constructs gateway-owned policy, provider state, networking, or the workload.
It reports bootstrap results after those components, the workload, and relay
endpoints are ready.
The gateway and supervisor must implement the same internal supervisor protocol
revision. Peers built before the handshake existed report revision zero and are
accepted for one release with a warning and a counter, because sandboxes keep
their supervisor binary until they are recreated. The gateway includes a configuration bootstrap when it accepts a
`ConnectSupervisor` session and can send complete component replacements on the
same stream after policy, settings, or provider state changes.
While polling remains authoritative, optional bootstrap construction has a
one-second budget. The gateway accepts the session without a bootstrap when
that budget expires, so slow credential backends do not block relay reconnects.
These payloads describe the latest effective state rather than
the mutation that produced it. The gateway assigns ordering sequences within
each session and component, while each snapshot retains its own content
@@ -537,8 +619,7 @@ Bootstrap components are independent read projections, not one atomic database
snapshot. The sandbox configuration carries the provider-environment revision
it was built against. The gateway retries bootstrap construction when that
revision does not match the provider snapshot. Later component updates and
owner reconciliation repair changes committed while the other projections
were being built.
polling repair changes committed while the other projections were being built.
Configuration delivery goes through a gateway-owned routing boundary rather
than exposing local supervisor channels to mutation handlers. The current
@@ -548,38 +629,21 @@ without changing publishers. Provider payloads can contain
credentials, so the gateway does not persist or render complete stream
messages in logs.
The supervisor applies bootstrap and live stream snapshots through shared
component runtime primitives. It reports the requested and
active revisions plus a component-specific outcome on `ConnectSupervisor`.
Sandbox-scoped policy results update only the matching policy-history row, so a
late result cannot mark a newer revision loaded. Explicit local policy remains
authoritative and produces a retained-local-override result. Other component
results persist only compact observed state: requested and active revisions,
outcome, effective source, observation time, and a bounded sanitized error.
Delivered snapshots, including provider credentials, are never persisted.
The gateway keeps one update in flight per session and component. It replaces
the pending snapshot when newer desired state arrives, validates the update ID,
component sequence, component, and requested revision on acknowledgement, then
sends the newest pending snapshot. A 30-second owner reconciliation pass
rebuilds current snapshots for active local sessions. Applied equality
revisions suppress unchanged delivery, while failed or timed-out delivery is
retried from current database state. Reconnect discards session delivery state
and starts with a fresh bootstrap.
Revision 2 does not poll configuration fetch APIs. Polling remains available
only to revision 1 and revision 0 supervisors during the mixed-version rollout.
The gateway serializes construction per sandbox and component, and coalesces
repeated mutations into the latest full snapshot. An enqueue result means only
that the local stream queue accepted the message. A bounded scope fanout scheduler
coalesces repeated workspace and global changes,
The supervisor currently parses and ignores stream-delivered configuration.
Polling remains the only path that changes runtime state and repairs dropped or
unavailable delivery. The gateway serializes construction per sandbox and
component, and coalesces repeated mutations into the latest full snapshot. An
enqueue result means only that the local stream queue accepted the message. A
bounded scope fanout scheduler coalesces repeated workspace and global changes,
and semaphores sized from the database pool bound delivery workers and snapshot
builds. Fanout waits for worker capacity before admitting each recipient, so a
fleet-wide change cannot create a fleet-sized task backlog or saturate the store
and credential backends. Snapshot construction has a deadline that starts once
a build holds a permit, and the gateway rejects encoded stream messages that
approach the transport decoder limit. Durable apply operations and final polling
removal remain separate follow-up work.
approach the transport decoder limit. A later migration will apply these
payloads directly and acknowledge their exact revisions before removing
supervisor polling. At that point, the gateway will require a valid bootstrap
before marking a session ready.
## Policy Revision Acknowledgement
@@ -592,11 +656,12 @@ If policy construction fails, it reports the captured revision as `FAILED` with
the original construction error. It never infers revision identity by comparing
policy structure.
Image-specific policy discovery and baseline enrichment can require one initial
gateway synchronization. The supervisor commits that repair before runtime
initialization, discards the mutation response, and reconnects so it installs
only the fresh authoritative stream bootstrap. Compatibility supervisors retain
the earlier enrichment and first-poll reconciliation path.
This holds even when the initial policy is enriched with baseline paths during
startup: the enriched revision the supervisor synced back to the gateway is the
revision it acknowledges, so a successfully constructed initial policy never
remains `Pending`. If the first poll returns a different revision, the supervisor
processes it through the normal reload path instead of treating it as already
loaded.
A newer sandbox-scoped revision can carry the same non-empty effective policy
hash as the currently loaded revision, for example when provenance changes
@@ -607,32 +672,43 @@ reconciliation succeeds. Global policies, local overrides, equal or older
versions, and different hashes do not use this shortcut. Success telemetry is
emitted only after the gateway accepts the resulting loaded-status report.
Revision 2 policy status is recorded from the correlated stream result. The
retained reporting RPC uses the same domain helper for compatibility
supervisors. Retryable legacy status delivery uses a FIFO background worker so
status endpoint outages do not block enforcement.
Policy status delivery uses a FIFO background worker. Retryable delivery
failures retain the ordered update and retry with capped exponential backoff;
terminal errors are logged and discarded. The outbox is nonblocking and does
not discard updates because of a fixed queue capacity, so status endpoint
outages cannot block policy polling, enforcement, settings, or provider
refreshes and cannot permanently lose the initial acknowledgement.
Only sandbox-scoped revisions (`PolicySource::Sandbox`, version greater than
zero) are acknowledged. Global policies and local-file development policies do
not use the sandbox revision API and produce no acknowledgement. When explicit
local Rego and data files are configured, the supervisor continues polling the
gateway for settings and provider refreshes only on the compatibility path; a
revision 2 supervisor receives those components on the stream and never
replaces the local OPA engine with a gateway policy revision.
zero) are acknowledged through the policy status API. The configuration stream
separately acknowledges complete snapshot application for every component.
Global policies and local-file development policies do not use the sandbox
revision API. When explicit local Rego and data files are provisioned into the
supervisor, streamed settings, middleware, and provider state still apply, but
gateway policy revisions never replace the local OPA engine. Workload image
files and environment variables do not configure the separately isolated
supervisor.
## Failure Behavior
- If a compatibility configuration poll fails, the sandbox keeps its
last-known-good policy.
- If compatibility config polling fails, the supervisor keeps its
last-known-good policy. Current protocol sessions use complete streamed
snapshots and reconnect with a fresh bootstrap.
- If a live policy or middleware-registry update is invalid, the supervisor
rejects the combined update and keeps the current runtime pair.
rejects the update and keeps the current runtime pair.
- If an operator-run middleware call fails, the selected config's `on_error`
behavior decides whether to deny the request or continue without that stage.
- Existing raw byte streams are connection scoped. Dynamic policy changes apply
to new connections or the next parsed HTTP request where the proxy can safely
re-evaluate.
- If the supervisor relay drops, the sandbox can keep running, but connect and
exec operations fail until the supervisor registers again.
- If the supervisor relay drops, the sandbox stops the canonical agent and exec
process groups, rejects new runtime operations, and closes mediated streams.
A replacement supervisor has 30 seconds to authenticate, replay the identical
attach, and reconfirm the boundary. Successful confirmation resumes the
process tree; otherwise the sandbox sends `SIGTERM`, waits the normal stop
grace period, sends `SIGKILL` to survivors, and makes the session terminal.
Explicit supervisor shutdown uses the same terminal transition and requires
an acknowledgement before treating the boundary as stopped.
- If the canonical main process exits, the supervisor durably reports the
normalized result immediately. A foreground create declares a one-shot main
attachment, so the supervisor accepts it even after a fast process exits,
@@ -645,3 +721,23 @@ replaces the local OPA engine with a gateway policy revision.
`Error/MainProcessFailed`. Infrastructure failures also use `Error`, with a
distinct condition reason and no fabricated canonical-process result. Runtime
restart policies must not replace the canonical process.
## Shared Boundary Primitives
`openshell-isolation-interface` owns the common boundary protocol and Linux
mechanisms. Drivers provide the protected transport and immutable resource
identity; they do not implement their own process or network protocol. All
remote traffic uses one mutually authenticated gRPC connection. Independent
streams carry process control, exec output, and TCP bytes; a persistent
`Mediate` stream carries DNS queries and supervisor-produced answers. There is
no alternate raw-TLS application protocol or general UDP framing.
The shared process-signal mediator resolves each positive target PID or TID to
its thread-group leader, excludes the sandbox leader, retains a pidfd, and sends
the signal through that descriptor. It never continues the original numeric-PID
syscall after inspection. This prevents TID aliases or PID reuse from turning an
agent signal into a signal to the sandbox. Ordinary mediated `kill` reports the
broker as its sender, not the original calling agent's `SI_USER` identity.
Queued signals preserve permitted application siginfo payloads; they cannot
forge kernel-generated or `SI_TKILL` codes. Programs requiring original sender
identity must account for this mediation boundary.
architecture/security-policy.md
+11 -7
View File
@@ -172,13 +172,15 @@ policy without provenance applies neither the raw-tunnel refusal nor the
WebSocket binary-frame refusal. The request-body backstop still applies, because
it keys off the presence of a secret resolver rather than endpoint provenance.
Two paths load a policy without provenance. A supervisor booting from a
container-image policy is a bounded window: that policy is resynchronized to the
gateway, which then serves a stamped effective policy. An explicit local Rego and
data override is permanent, because gateway revisions are observed for settings
and providers but never replace the local policy. When that override is combined
with injected provider credentials, the supervisor emits a high-severity
detection finding at startup naming the inactive controls.
Two supervisor-local paths load a policy without provenance. A supervisor
booting from an explicitly provisioned policy file has a bounded window before
that policy is resynchronized to the gateway, which then serves a stamped
effective policy. An explicit supervisor Rego and data override is permanent,
because gateway revisions are observed for settings and providers but never
replace the local policy. Workload-image files and environment variables cannot
configure the separately isolated supervisor. When a supervisor override is
combined with injected provider credentials, the supervisor emits a
high-severity detection finding at startup naming the inactive controls.
## Live Updates
@@ -189,6 +191,8 @@ supervisor polls for config revisions and attempts to load new dynamic policy
into the in-process OPA engine; CLI reads of the latest sandbox policy use the
same effective configuration path.
The OPA loader checks the object and list shapes of raw policy data before injecting runtime fields, normalizing values, or expanding access presets. It rejects the first malformed container with a fixed structural error that excludes authored keys and values. This check preserves valid versionless OPA data and runtime-only fields. A rejected OPA engine reload leaves that engine's installed policy, generation, and decisions unchanged; the supervisor separately applies its configured runtime rejection mode.
The supervisor validates complete effective policy generations before
activation. Overlapping endpoint selectors may contribute request allow and
deny rules only when their connection and request-processing metadata agree;
architecture/windows-msvc-build.md
+24 -11
View File
@@ -49,6 +49,10 @@ domain sockets. Their libraries remain in the gateway dependency graph, so the
gateway's credential-driver configuration and in-process behavior still compile
on Windows.
The standalone sandbox and supervisor runtimes are Unix-only and are excluded
as top-level Windows workspace targets. The MXC driver links only the
cross-platform supervisor network library needed by its host egress proxy.
| Driver | Windows build behavior | Runtime behavior |
|---|---|---|
| Docker | Driver crate excluded; gateway registration stub retained. | Gateway construction returns unsupported. |
@@ -108,15 +112,24 @@ async functions caused by cfg-gated Windows stubs. Repository-wide pre-commit
skips only Linux-specific installer, build-environment shell-helper, and
packaging-asset tests; its
cross-platform Python, Markdown, license, and documentation checks still run.
Tracked Cargo lockfiles are checked natively through PowerShell. Deterministic
gateway parity uses Git for Windows Bash with temporary, checkout-scoped Python
launchers. The TypeScript SDK uses Windows protobuf plugin paths and x64 Biome
under emulation on ARM64, while its test binding follows Node's architecture
and the locked Rolldown version. Go tests retain race coverage wherever the
toolchain supports it; POSIX permission-bit checks are not Windows ACL tests.
Test tasks require the Rust target architecture to match the Windows host, so
an ARM64 test result is native coverage rather than x64 emulation coverage.
By default it enables bundled Z3 for reproducible Windows builds. `z3-sys`
builds Z3 from source and links it into the release binaries, so copying
`openshell-gateway.exe` and `openshell.exe` does not create an undeclared
`libz3.dll` runtime dependency. When `Z3_LIBRARY_PATH_OVERRIDE` points at a
directory containing `libz3.lib`, the wrapper uses that system Z3 instead and
requires `Z3_SYS_Z3_HEADER` to point at the full path to `z3.h`. Binaries built
against a dynamic system Z3 must be deployed with the matching `libz3.dll`.
By default it enables the `z3-sys` prebuilt-release feature and pins Z3 4.16.0.
On a clean target directory, `z3-sys` downloads the official static library for
the selected Windows architecture instead of compiling Z3 through
CMake/MSBuild. GitHub Actions supplies its read-only workflow token for the
release lookup, and the Cargo target cache preserves the extracted library for
subsequent runs. When
`Z3_LIBRARY_PATH_OVERRIDE` points at a directory containing `libz3.lib`, the
wrapper uses that system Z3 instead and requires `Z3_SYS_Z3_HEADER` to point at
the full path to `z3.h`. Local clean builds use the unauthenticated GitHub API
unless `READ_ONLY_GITHUB_TOKEN` is set.
GitHub Actions layers the Cargo target cache with sccache's GitHub Actions
backend. The target cache lets Cargo skip intact dependency builds; sccache
@@ -136,10 +149,10 @@ Spectre-mitigated libraries, host-native Clang tools, CMake tools, and an
ARM64-capable Windows SDK. Clang provides `libclang.dll` for `bindgen` and
`clang-cl.exe` for ARM64 crypto dependencies. During x64-to-ARM64 check/build,
the wrapper discovers and adds the Visual Studio-bundled Ninja to `PATH` for
native dependencies. Bundled Z3 uses native MSVC `cl.exe` with the Visual
Studio generator, while the crypto crates select `clang-cl`. Artifact hashing
uses .NET SHA256 directly because module autoloading in the mise-launched
Windows PowerShell process is not guaranteed.
native dependencies. Z3 uses the official prebuilt ARM64 static library, so it
does not inherit compiler settings from those native dependencies. Artifact
hashing uses .NET SHA256 directly because module autoloading in the
mise-launched Windows PowerShell process is not guaranteed.
The wrapper defaults Cargo compilation to four jobs. Set
`OPENSHELL_WINDOWS_BUILD_JOBS` to a positive integer to override that limit.
crates/openshell-binary-identity/Cargo.toml
+18
View File
@@ -0,0 +1,18 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
[package]
name = "openshell-binary-identity"
description = "Trusted executable identity resolution for OpenShell isolation backends"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
repository.workspace = true
[dependencies]
openshell-isolation-interface = { path = "../openshell-isolation-interface" }
sha2 = { workspace = true }
[lints]
workspace = true
crates/openshell-binary-identity/README.md
+17
View File
@@ -0,0 +1,17 @@
# Binary identity
`openshell-binary-identity` provides shared executable-identity resolution for
RFC 0012 isolation backends. Runtime-specific observers remain in their backend:
Docker obtains an authoritative thread ID from seccomp notification, while the
co-located Linux path maps an accepted socket to its owning processes.
Given an authoritative Linux PID and an optional trusted process-tree root, the
crate reads the executable path from procfs, hashes the live `/proc/<pid>/exe`
object, and collects bounded executable ancestry and diagnostic command-line
paths. Resolution failures are returned as `ResolveError` so the caller can
deny the associated connection.
The crate does not intercept connections, authenticate remote observers, or
evaluate policy. The isolation backend remains responsible for binding the
resolved identity to the active boundary and exact accepted connection before
constructing `MediatedConnection`.
crates/openshell-binary-identity/src/lib.rs
+475
View File
@@ -0,0 +1,475 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
//! Shared executable-identity resolution for RFC 0012 isolation backends.
//!
//! Runtime-specific observation remains inside each isolation backend. Once an
//! observer has an authoritative PID in its procfs view, this crate
//! canonicalizes the executable path, hashes the live executable object, and
//! collects its process ancestry. Backends bind the returned identity to the
//! intercepted connection before constructing a `MediatedConnection`.
#[cfg(target_os = "linux")]
use openshell_isolation_interface::contract::Sha256Digest;
use openshell_isolation_interface::contract::{BinaryIdentity, ResolveError};
#[cfg(target_os = "linux")]
use std::collections::HashMap;
#[cfg(target_os = "linux")]
use std::sync::{Arc, Mutex};
#[cfg(target_os = "linux")]
const EXECUTABLE_DIGEST_CACHE_CAPACITY: usize = 1_024;
/// Resolves executable identity from a Linux procfs process identifier.
///
/// The configured scope bounds ancestry and cmdline collection to the observed
/// PID namespace or a known workload process tree.
#[derive(Clone, Debug)]
pub struct ProcfsIdentityResolver {
ancestry_scope: AncestryScope,
#[cfg(target_os = "linux")]
cache: Arc<Mutex<HashMap<ExecutableCacheKey, Sha256Digest>>>,
}
#[derive(Clone, Copy, Debug)]
enum AncestryScope {
PidNamespace,
ProcessTree(u32),
}
impl Default for ProcfsIdentityResolver {
fn default() -> Self {
Self::for_pid_namespace()
}
}
impl ProcfsIdentityResolver {
/// Build a resolver that discovers a nested PID namespace's init process
/// and never reports host-runtime ancestors outside that namespace.
#[must_use]
pub fn for_pid_namespace() -> Self {
Self {
ancestry_scope: AncestryScope::PidNamespace,
#[cfg(target_os = "linux")]
cache: Arc::new(Mutex::new(HashMap::new())),
}
}
/// Build a resolver bounded by the workload's trusted process-tree root.
#[must_use]
pub fn for_process_tree(ancestor_root: u32) -> Self {
Self {
ancestry_scope: AncestryScope::ProcessTree(ancestor_root),
#[cfg(target_os = "linux")]
cache: Arc::new(Mutex::new(HashMap::new())),
}
}
/// Resolve the identity for an authoritative process ID.
pub fn resolve(&self, pid: u32) -> Result<BinaryIdentity, ResolveError> {
#[cfg(target_os = "linux")]
{
let ancestor_root = match self.ancestry_scope {
AncestryScope::PidNamespace => nested_pid_namespace_init(pid),
AncestryScope::ProcessTree(root) => Some(root),
};
resolve_linux_process(pid, ancestor_root, &self.cache)
}
#[cfg(not(target_os = "linux"))]
{
match self.ancestry_scope {
AncestryScope::PidNamespace => {}
AncestryScope::ProcessTree(ancestor_root) => {
let _ = ancestor_root;
}
}
let _ = pid;
Err(ResolveError::Failed(
"procfs binary identity is only available on Linux".to_string(),
))
}
}
}
#[cfg(target_os = "linux")]
fn resolve_linux_process(
pid: u32,
ancestor_root: Option<u32>,
cache: &Mutex<HashMap<ExecutableCacheKey, Sha256Digest>>,
) -> Result<BinaryIdentity, ResolveError> {
let (snapshot, mut executable) = open_process_snapshot(pid)?;
let binary_path = snapshot.binary_path.clone();
let executable_key = snapshot.executable_cache_key();
let cached_digest = cached_executable_digest(cache, executable_key);
let binary_digest = cached_digest.map_or_else(|| hash_executable(pid, &mut executable), Ok)?;
let ancestor_processes = collect_ancestor_processes(&snapshot, ancestor_root);
let ancestors = ancestor_processes
.iter()
.map(|snapshot| snapshot.binary_path.clone())
.collect::<Vec<_>>();
let mut excluded_paths = ancestors.clone();
excluded_paths.push(binary_path.clone());
let cmdline_paths = cmdline_absolute_paths(&snapshot.cmdline)
.into_iter()
.chain(
ancestor_processes
.iter()
.flat_map(|snapshot| cmdline_absolute_paths(&snapshot.cmdline)),
)
.filter(|path| !excluded_paths.contains(path))
.fold(Vec::new(), |mut paths, path| {
if !paths.contains(&path) {
paths.push(path);
}
paths
});
validate_process_snapshot(pid, &snapshot)?;
for ancestor in &ancestor_processes {
validate_process_snapshot(ancestor.pid, ancestor)?;
}
if cached_digest.is_none() {
cache_executable_digest(cache, executable_key, binary_digest);
}
Ok(BinaryIdentity {
binary_path,
binary_digest: Some(binary_digest),
ancestors,
cmdline_paths,
})
}
#[cfg(target_os = "linux")]
#[derive(Debug, PartialEq, Eq)]
struct ProcessSnapshot {
pid: u32,
parent_pid: u32,
binary_path: std::path::PathBuf,
executable_device: u64,
executable_inode: u64,
executable_size: u64,
executable_mtime: i64,
executable_mtime_nsec: i64,
executable_ctime: i64,
executable_ctime_nsec: i64,
start_time: u64,
cmdline: Vec<u8>,
}
#[cfg(target_os = "linux")]
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
struct ExecutableCacheKey {
device: u64,
inode: u64,
size: u64,
mtime: i64,
mtime_nsec: i64,
ctime: i64,
ctime_nsec: i64,
}
#[cfg(target_os = "linux")]
impl ProcessSnapshot {
fn executable_cache_key(&self) -> ExecutableCacheKey {
ExecutableCacheKey {
device: self.executable_device,
inode: self.executable_inode,
size: self.executable_size,
mtime: self.executable_mtime,
mtime_nsec: self.executable_mtime_nsec,
ctime: self.executable_ctime,
ctime_nsec: self.executable_ctime_nsec,
}
}
}
#[cfg(target_os = "linux")]
fn cached_executable_digest(
cache: &Mutex<HashMap<ExecutableCacheKey, Sha256Digest>>,
key: ExecutableCacheKey,
) -> Option<Sha256Digest> {
cache
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.get(&key)
.copied()
}
#[cfg(target_os = "linux")]
fn cache_executable_digest(
cache: &Mutex<HashMap<ExecutableCacheKey, Sha256Digest>>,
key: ExecutableCacheKey,
digest: Sha256Digest,
) {
let mut cache = cache
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if cache.len() >= EXECUTABLE_DIGEST_CACHE_CAPACITY {
cache.clear();
}
cache.insert(key, digest);
}
#[cfg(target_os = "linux")]
fn open_process_snapshot(pid: u32) -> Result<(ProcessSnapshot, std::fs::File), ResolveError> {
use std::os::unix::fs::MetadataExt as _;
let path = format!("/proc/{pid}/exe");
let binary_path = executable_path(pid)?;
let executable = std::fs::File::open(&path)
.map_err(|error| ResolveError::Failed(format!("open {path}: {error}")))?;
let metadata = executable
.metadata()
.map_err(|error| ResolveError::Failed(format!("stat {path}: {error}")))?;
let (parent_pid, start_time) = process_stat(pid)?;
let snapshot = ProcessSnapshot {
pid,
parent_pid,
binary_path,
executable_device: metadata.dev(),
executable_inode: metadata.ino(),
executable_size: metadata.size(),
executable_mtime: metadata.mtime(),
executable_mtime_nsec: metadata.mtime_nsec(),
executable_ctime: metadata.ctime(),
executable_ctime_nsec: metadata.ctime_nsec(),
start_time,
cmdline: read_process_cmdline(pid)?,
};
validate_process_snapshot(pid, &snapshot)?;
Ok((snapshot, executable))
}
#[cfg(target_os = "linux")]
fn validate_process_snapshot(pid: u32, expected: &ProcessSnapshot) -> Result<(), ResolveError> {
use std::os::unix::fs::MetadataExt as _;
let path = format!("/proc/{pid}/exe");
let metadata = std::fs::metadata(&path)
.map_err(|error| ResolveError::Failed(format!("stat {path}: {error}")))?;
let (parent_pid, start_time) = process_stat(pid)?;
let current = ProcessSnapshot {
pid,
parent_pid,
binary_path: executable_path(pid)?,
executable_device: metadata.dev(),
executable_inode: metadata.ino(),
executable_size: metadata.size(),
executable_mtime: metadata.mtime(),
executable_mtime_nsec: metadata.mtime_nsec(),
executable_ctime: metadata.ctime(),
executable_ctime_nsec: metadata.ctime_nsec(),
start_time,
cmdline: read_process_cmdline(pid)?,
};
if &current == expected {
Ok(())
} else {
Err(ResolveError::Failed(format!(
"process {pid} changed while its executable identity was collected"
)))
}
}
#[cfg(target_os = "linux")]
fn process_stat(pid: u32) -> Result<(u32, u64), ResolveError> {
let path = format!("/proc/{pid}/stat");
let stat = std::fs::read_to_string(&path)
.map_err(|error| ResolveError::Failed(format!("read {path}: {error}")))?;
let fields = stat
.rsplit_once(") ")
.map(|(_, fields)| fields)
.ok_or_else(|| ResolveError::Failed(format!("parse {path}: missing command field")))?;
let mut fields = fields.split_whitespace();
let _state = fields.next();
let parent_pid = fields
.next()
.ok_or_else(|| ResolveError::Failed(format!("parse {path}: missing parent PID")))?
.parse()
.map_err(|error| ResolveError::Failed(format!("parse {path} parent PID: {error}")))?;
let start_time = fields
.nth(17)
.ok_or_else(|| ResolveError::Failed(format!("parse {path}: missing start time")))?
.parse()
.map_err(|error| ResolveError::Failed(format!("parse {path} start time: {error}")))?;
Ok((parent_pid, start_time))
}
#[cfg(target_os = "linux")]
fn read_process_cmdline(pid: u32) -> Result<Vec<u8>, ResolveError> {
let path = format!("/proc/{pid}/cmdline");
std::fs::read(&path).map_err(|error| ResolveError::Failed(format!("read {path}: {error}")))
}
#[cfg(target_os = "linux")]
fn executable_path(pid: u32) -> Result<std::path::PathBuf, ResolveError> {
use std::ffi::OsString;
use std::io::ErrorKind;
use std::os::unix::ffi::{OsStrExt as _, OsStringExt as _};
const DELETED_SUFFIX: &[u8] = b" (deleted)";
let link = format!("/proc/{pid}/exe");
let target = std::fs::read_link(&link)
.map_err(|error| ResolveError::Failed(format!("read {link}: {error}")))?;
let target_missing =
matches!(std::fs::metadata(&target), Err(error) if error.kind() == ErrorKind::NotFound);
let bytes = target.as_os_str().as_bytes();
if target_missing && bytes.ends_with(DELETED_SUFFIX) {
let stripped = bytes[..bytes.len() - DELETED_SUFFIX.len()].to_vec();
return Ok(std::path::PathBuf::from(OsString::from_vec(stripped)));
}
Ok(target)
}
#[cfg(target_os = "linux")]
fn hash_executable(pid: u32, executable: &mut std::fs::File) -> Result<Sha256Digest, ResolveError> {
use sha2::{Digest as _, Sha256};
use std::io::Read as _;
let path = format!("/proc/{pid}/exe");
let mut digest = Sha256::new();
let mut buffer = [0_u8; 8 * 1024];
loop {
let length = executable
.read(&mut buffer)
.map_err(|error| ResolveError::Failed(format!("hash {path}: {error}")))?;
if length == 0 {
break;
}
digest.update(&buffer[..length]);
}
format!("{:x}", digest.finalize()).parse()
}
#[cfg(target_os = "linux")]
fn collect_ancestor_processes(
process: &ProcessSnapshot,
ancestor_root: Option<u32>,
) -> Vec<ProcessSnapshot> {
const MAX_DEPTH: usize = 64;
if ancestor_root == Some(process.pid) {
return Vec::new();
}
let mut ancestors = Vec::new();
let mut parent = process.parent_pid;
for _ in 0..MAX_DEPTH {
if parent == 0
|| ancestors
.iter()
.any(|current: &ProcessSnapshot| current.pid == parent)
{
break;
}
// PID 1 is host or guest init rather than workload ancestry unless it
// is the explicitly supplied process-tree root.
if parent == 1 && ancestor_root != Some(1) {
break;
}
let Ok((snapshot, _executable)) = open_process_snapshot(parent) else {
break;
};
let next_parent = snapshot.parent_pid;
ancestors.push(snapshot);
if ancestor_root == Some(parent) || parent == 1 {
break;
}
parent = next_parent;
}
ancestors
}
#[cfg(target_os = "linux")]
fn parent_pid(pid: u32) -> Option<u32> {
std::fs::read_to_string(format!("/proc/{pid}/status"))
.ok()?
.lines()
.find_map(|line| line.strip_prefix("PPid:"))?
.trim()
.parse()
.ok()
}
#[cfg(target_os = "linux")]
fn nested_pid_namespace_init(pid: u32) -> Option<u32> {
const MAX_DEPTH: usize = 64;
let mut current = pid;
for _ in 0..MAX_DEPTH {
if namespace_pid(current) == Some(1) {
// Host PID 1 is outside every workload. A nested namespace init
// has a distinct host PID and is a valid workload ancestry root.
return (current != 1).then_some(current);
}
current = parent_pid(current).filter(|parent| *parent > 0 && *parent != current)?;
}
None
}
#[cfg(target_os = "linux")]
fn namespace_pid(pid: u32) -> Option<u32> {
std::fs::read_to_string(format!("/proc/{pid}/status"))
.ok()?
.lines()
.find_map(|line| line.strip_prefix("NSpid:"))?
.split_whitespace()
.next_back()?
.parse()
.ok()
}
#[cfg(target_os = "linux")]
fn cmdline_absolute_paths(cmdline: &[u8]) -> Vec<std::path::PathBuf> {
cmdline
.split(|byte| *byte == 0)
.filter(|argument| argument.first() == Some(&b'/'))
.map(|argument| std::path::PathBuf::from(String::from_utf8_lossy(argument).into_owned()))
.collect()
}
#[cfg(all(test, target_os = "linux"))]
mod tests {
use super::*;
#[cfg(target_os = "linux")]
#[test]
fn resolver_cache_is_owned_and_only_explicit_clones_share_it() {
let first = ProcfsIdentityResolver::for_pid_namespace();
let shared = first.clone();
let separate = ProcfsIdentityResolver::for_pid_namespace();
assert!(Arc::ptr_eq(&first.cache, &shared.cache));
assert!(!Arc::ptr_eq(&first.cache, &separate.cache));
first.resolve(std::process::id()).unwrap();
assert!(!shared.cache.lock().unwrap().is_empty());
assert!(separate.cache.lock().unwrap().is_empty());
}
#[test]
fn resolves_current_process_from_live_executable() {
let identity = ProcfsIdentityResolver::for_pid_namespace()
.resolve(std::process::id())
.expect("resolve current process");
assert!(identity.binary_path.is_absolute());
assert!(identity.binary_digest.is_some());
}
#[test]
fn process_tree_root_does_not_escape_into_host_ancestry() {
let pid = std::process::id();
let identity = ProcfsIdentityResolver::for_process_tree(pid)
.resolve(pid)
.expect("resolve process-tree root");
assert!(identity.ancestors.is_empty());
}
}
crates/openshell-cli/src/main.rs
+30 -5
View File
@@ -1372,7 +1372,7 @@ enum SandboxCommands {
no_tty: bool,
/// Start the canonical main process without attaching to it.
#[arg(long, conflicts_with_all = ["editor", "no_keep"])]
#[arg(long, conflicts_with = "editor")]
detach: bool,
/// Auto-create missing providers from local credentials.
@@ -5461,10 +5461,35 @@ mod tests {
}
#[test]
fn sandbox_create_detach_rejects_ephemeral_sandbox() {
let result =
Cli::try_parse_from(["openshell", "sandbox", "create", "--detach", "--no-keep"]);
assert!(result.is_err());
fn sandbox_create_detach_accepts_ephemeral_sandbox() {
let cli = Cli::try_parse_from([
"openshell",
"sandbox",
"create",
"--detach",
"--no-keep",
"--",
"worker",
])
.expect("detached ephemeral sandbox should parse");
match cli.command {
Some(Commands::Sandbox {
command:
Some(SandboxCommands::Create {
detach,
no_keep,
command,
..
}),
..
}) => {
assert!(detach);
assert!(no_keep);
assert_eq!(command, ["worker"]);
}
other => panic!("expected SandboxCommands::Create, got: {other:?}"),
}
}
#[test]
crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs
+40
View File
@@ -1700,6 +1700,46 @@ async fn detached_command_does_not_declare_main_process_attachment() {
assert!(!requests[0].await_main_process_attachment);
}
#[tokio::test]
async fn detached_ephemeral_command_delegates_cleanup_to_gateway() {
let server = run_server().await;
let fake_ssh_dir = tempfile::tempdir().unwrap();
let xdg_dir = tempfile::tempdir().unwrap();
let _env = test_env(&fake_ssh_dir, &xdg_dir);
let tls = test_tls(&server);
install_fake_ssh(&fake_ssh_dir);
run::sandbox_create(
&server.endpoint,
"openshell",
run::SandboxCreateConfig {
name: Some("detached-ephemeral-main"),
keep: false,
command: &["worker".into()],
detach: true,
..test_config()
},
"default",
&tls,
)
.await
.expect("detached ephemeral sandbox create should succeed");
let requests = create_requests(&server).await;
assert!(!requests[0].await_main_process_attachment);
assert_eq!(
requests[0]
.annotations
.get("openshell.nvidia.com/retention")
.map(String::as_str),
Some("ephemeral")
);
assert!(
deleted_names(&server).await.is_empty(),
"the gateway owns cleanup after a detached canonical process exits"
);
}
#[tokio::test]
async fn sandbox_create_sends_driver_config_json() {
let server = run_server().await;
crates/openshell-conformance/src/lib.rs
+16
View File
@@ -288,6 +288,22 @@ impl OpenShellRunner {
))
}
/// Uses the candidate CLI selected by the archive test runner.
///
/// Archive-based tests set `OPENSHELL_BIN` to the candidate artifact
/// installed in the guest. Requiring it here prevents a test from silently
/// resolving a different `openshell` binary from `PATH`.
pub fn from_env(scenario: &str) -> Result<Self, RunnerError> {
let binary = std::env::var_os("OPENSHELL_BIN")
.map(PathBuf::from)
.ok_or_else(|| {
RunnerError::BinaryUnavailable(
"OPENSHELL_BIN must name the candidate openshell CLI".to_string(),
)
})?;
Self::with_binary(binary, scenario)
}
/// Creates a runner with an injected executor. This is useful for harness tests.
pub fn with_executor(cli: Arc<dyn CliExecutor>, scenario: &str) -> Self {
Self {
crates/openshell-core/Cargo.toml
+6
View File
@@ -18,6 +18,7 @@ prost = { workspace = true }
prost-types = { workspace = true }
tonic = { workspace = true, features = ["channel", "tls-aws-lc"] }
tonic-prost = { workspace = true }
tonic-types = { workspace = true }
tokio = { workspace = true }
tokio-stream = { workspace = true }
thiserror = { workspace = true }
@@ -27,10 +28,13 @@ serde_json = { workspace = true }
sha2 = { workspace = true }
tracing = { workspace = true }
url = { workspace = true }
uuid = { workspace = true }
ipnet = "2"
rustls = { workspace = true }
rustls-pemfile = { workspace = true }
base64 = { workspace = true }
jsonwebtoken = { workspace = true, optional = true }
zeroize = { workspace = true, optional = true }
chrono = { version = "0.4", default-features = false, features = ["clock", "std"], optional = true }
reqwest = { workspace = true, features = ["blocking", "rustls-tls-native-roots-no-provider"], optional = true }
tar = { version = "0.4", optional = true }
@@ -49,6 +53,8 @@ driver-extraction = ["dep:tar", "dep:tempfile"]
telemetry = ["dep:reqwest", "dep:chrono", "reqwest?/blocking"]
## OAuth2 token request helpers used by supervisor and gateway.
oauth = ["dep:reqwest"]
## Strict Ed25519 JWT issuance and verification for authenticated sandbox sessions.
jwt = ["dep:jsonwebtoken", "dep:zeroize"]
[build-dependencies]
tonic-prost-build = { workspace = true }
crates/openshell-core/src/config.rs
+31 -2
View File
@@ -86,6 +86,18 @@ impl FromStr for PolicyValidationFailureMode {
/// Default OCI repository for the supervisor image (no tag).
pub const DEFAULT_SUPERVISOR_IMAGE_REPO: &str = "ghcr.io/nvidia/openshell/supervisor";
/// Default OCI repository for the sandbox runtime image (no tag).
pub const DEFAULT_SANDBOX_RUNTIME_IMAGE_REPO: &str = "ghcr.io/nvidia/openshell/sandbox";
/// Return the default sandbox runtime image reference with a version-pinned tag.
#[must_use]
pub fn default_sandbox_runtime_image() -> String {
format!(
"{DEFAULT_SANDBOX_RUNTIME_IMAGE_REPO}:{}",
default_supervisor_image_tag()
)
}
/// Return the default supervisor image reference with a version-pinned tag.
#[must_use]
pub fn default_supervisor_image() -> String {
@@ -319,9 +331,21 @@ pub struct TlsConfig {
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct OidcConfig {
/// OIDC issuer URL (e.g., `http://localhost:8180/realms/openshell`).
/// OIDC issuer URL (e.g., `https://idp.example.com/realms/openshell`).
pub issuer: String,
/// Permit cleartext OIDC metadata and JWKS requests to numeric loopback
/// addresses. This is a development-only escape hatch and never permits
/// cleartext requests to hostnames or non-loopback addresses.
#[serde(default)]
pub dangerously_allow_insecure_http: bool,
/// Additional origins from which JWKS may be loaded. Entries must be
/// origins such as `https://www.googleapis.com`, without a path, query,
/// credentials, or fragment. The issuer origin is always allowed.
#[serde(default)]
pub jwks_allowed_origins: Vec<String>,
/// Expected audience (`aud`) claim. Typically the OIDC client ID.
pub audience: String,
@@ -1539,10 +1563,15 @@ mod tests {
#[test]
fn default_supervisor_image_is_version_pinned() {
use super::default_supervisor_image;
use super::{default_sandbox_runtime_image, default_supervisor_image};
let image = default_supervisor_image();
assert!(image.starts_with("ghcr.io/nvidia/openshell/supervisor:"));
let tag = image.rsplit_once(':').unwrap().1;
assert!(!tag.is_empty());
let sandbox_image = default_sandbox_runtime_image();
assert!(sandbox_image.starts_with("ghcr.io/nvidia/openshell/sandbox:"));
let sandbox_tag = sandbox_image.rsplit_once(':').unwrap().1;
assert!(!sandbox_tag.is_empty());
}
}
crates/openshell-core/src/driver_utils.rs
+16 -13
View File
@@ -7,10 +7,10 @@ use std::path::{Path, PathBuf};
use crate::proto::compute::v1::DriverSandbox;
/// Built-in sandbox network topologies used to derive a callback endpoint
/// Built-in sandbox network callback routes used to derive a callback endpoint
/// when an operator does not configure a per-driver `grpc_endpoint` override.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum GatewayCallbackTopology {
pub enum GatewayCallbackRoute {
/// A Docker container reaches the host through Docker's gateway alias.
Docker,
/// A Podman container reaches the host through Podman's gateway alias.
@@ -19,41 +19,41 @@ pub enum GatewayCallbackTopology {
Vm,
}
/// Build the endpoint a sandbox uses to call its gateway for a known topology.
/// Build the endpoint a sandbox uses to call its gateway for a known route.
///
/// The result is deliberately derived by the gateway rather than baked into
/// individual driver defaults. A configured `grpc_endpoint` remains an
/// operator override for remote or non-standard deployments.
#[must_use]
pub fn gateway_callback_endpoint(
topology: GatewayCallbackTopology,
route: GatewayCallbackRoute,
gateway_port: u16,
gateway_tls_enabled: bool,
) -> String {
let scheme = if gateway_tls_enabled { "https" } else { "http" };
let host = match topology {
GatewayCallbackTopology::Docker | GatewayCallbackTopology::Vm => "host.openshell.internal",
GatewayCallbackTopology::Podman => "host.containers.internal",
let host = match route {
GatewayCallbackRoute::Docker | GatewayCallbackRoute::Vm => "host.openshell.internal",
GatewayCallbackRoute::Podman => "host.containers.internal",
};
format!("{scheme}://{host}:{gateway_port}")
}
#[cfg(test)]
mod callback_endpoint_tests {
use super::{GatewayCallbackTopology, gateway_callback_endpoint};
use super::{GatewayCallbackRoute, gateway_callback_endpoint};
#[test]
fn derives_endpoint_for_each_builtin_topology() {
fn derives_endpoint_for_each_builtin_route() {
assert_eq!(
gateway_callback_endpoint(GatewayCallbackTopology::Docker, 17670, false),
gateway_callback_endpoint(GatewayCallbackRoute::Docker, 17670, false),
"http://host.openshell.internal:17670"
);
assert_eq!(
gateway_callback_endpoint(GatewayCallbackTopology::Podman, 17670, true),
gateway_callback_endpoint(GatewayCallbackRoute::Podman, 17670, true),
"https://host.containers.internal:17670"
);
assert_eq!(
gateway_callback_endpoint(GatewayCallbackTopology::Vm, 17670, true),
gateway_callback_endpoint(GatewayCallbackRoute::Vm, 17670, true),
"https://host.openshell.internal:17670"
);
}
@@ -134,7 +134,10 @@ pub const CONDITION_STOPPED: &str = "ContainerStopped";
/// All compute drivers must launch this binary as the container entrypoint to
/// start the sandboxed environment. The value must be kept in sync with the
/// path used when building the `openshell-sandbox` image layer.
pub const SUPERVISOR_IMAGE_BINARY_PATH: &str = "/openshell-sandbox";
pub const SANDBOX_RUNTIME_IMAGE_BINARY_PATH: &str = "/openshell-sandbox";
/// Legacy name for [`SANDBOX_RUNTIME_IMAGE_BINARY_PATH`].
pub const SUPERVISOR_IMAGE_BINARY_PATH: &str = SANDBOX_RUNTIME_IMAGE_BINARY_PATH;
/// Directory inside sandbox containers where the supervisor binary is mounted.
///
crates/openshell-core/src/grpc_client.rs
+133 -18
View File
@@ -72,6 +72,9 @@ static TOKEN_INIT_LOCK: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new((
/// One-shot guard so the renewal loop spawns at most once per process.
static REFRESH_SPAWNED: OnceLock<()> = OnceLock::new();
#[cfg(feature = "jwt")]
static SANDBOX_BEARER_SLOT: OnceLock<crate::jwt::SessionBearerTokenSlot> = OnceLock::new();
#[derive(Clone, Debug)]
enum RefreshMode {
GatewayJwt(TokenSource),
@@ -84,16 +87,64 @@ struct AcquiredToken {
}
fn install_token_slot(token: &str) -> Result<TokenSlot> {
let bearer = AsciiMetadataValue::try_from(format!("Bearer {token}"))
let bearer = validate_gateway_bearer(token)?;
Ok(install_validated_token_slot(bearer))
}
fn validate_gateway_bearer(token: &str) -> Result<AsciiMetadataValue> {
AsciiMetadataValue::try_from(format!("Bearer {token}"))
.into_diagnostic()
.wrap_err("sandbox JWT contained characters not valid for a header value")?;
.wrap_err("sandbox JWT contained characters not valid for a header value")
}
fn install_validated_token_slot(bearer: AsciiMetadataValue) -> TokenSlot {
if let Some(existing) = TOKEN_SLOT.get() {
*existing.write().expect("token slot poisoned") = bearer;
return Ok(existing.clone());
*existing
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner) = bearer;
return existing.clone();
}
let slot: TokenSlot = Arc::new(RwLock::new(bearer));
let _ = TOKEN_SLOT.set(slot.clone());
Ok(TOKEN_SLOT.get().cloned().unwrap_or(slot))
TOKEN_SLOT.get().cloned().unwrap_or(slot)
}
#[cfg(feature = "jwt")]
struct ValidatedSandboxRefresh {
token: crate::jwt::SecretJwt,
expires_at: i64,
credential_epoch: crate::jwt::CredentialEpoch,
}
#[cfg(feature = "jwt")]
fn validate_sandbox_refresh(
response: &crate::proto::RefreshSandboxTokenResponse,
) -> std::result::Result<ValidatedSandboxRefresh, crate::jwt::SessionJwtError> {
let token = crate::jwt::SecretJwt::parse(response.sandbox_token.clone())?;
let credential_epoch = crate::jwt::CredentialEpoch::new(response.credential_epoch)?;
let expires_at = response.sandbox_expires_at_ms / 1000;
crate::jwt::SessionBearerTokenSlot::new(token.clone(), expires_at, credential_epoch)?;
Ok(ValidatedSandboxRefresh {
token,
expires_at,
credential_epoch,
})
}
/// Install the gateway-session credential supplied in trusted supervisor
/// launch state before any gateway client is constructed.
#[cfg(feature = "jwt")]
pub fn install_supervisor_auth_bundle(
bundle: &crate::jwt::SupervisorAuthBundle,
) -> Result<crate::jwt::SessionBearerTokenSlot> {
install_token_slot(bundle.gateway_token.expose_secret())?;
let _ = TOKEN_REFRESH_MODE.set(RefreshMode::GatewayJwt(TokenSource::File));
let slot = bundle
.sandbox_bearer_slot()
.into_diagnostic()
.wrap_err("invalid Sandbox Protocol credential")?;
let _ = SANDBOX_BEARER_SLOT.set(slot.clone());
Ok(SANDBOX_BEARER_SLOT.get().cloned().unwrap_or(slot))
}
/// gRPC interceptor that injects `authorization: Bearer <token>` on every
@@ -362,16 +413,38 @@ async fn refresh_token_loop(
.await
{
Ok(resp) => {
let new_token = resp.into_inner().token;
match AsciiMetadataValue::try_from(format!("Bearer {new_token}")) {
Ok(value) => {
if let Ok(mut guard) = slot.write() {
*guard = value;
info!("renewed gateway sandbox JWT in-place");
}
let response = resp.into_inner();
#[cfg(feature = "jwt")]
let sandbox_refresh = match validate_sandbox_refresh(&response) {
Ok(refresh) => refresh,
Err(error) => {
warn!(%error, "gateway returned an invalid Sandbox Protocol credential");
continue;
}
Err(e) => warn!(error = %e, "refreshed JWT contained invalid header bytes"),
};
let gateway_bearer = match validate_gateway_bearer(&response.token) {
Ok(value) => value,
Err(error) => {
warn!(%error, "refreshed JWT contained invalid header bytes");
continue;
}
};
*slot
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner) = gateway_bearer;
#[cfg(feature = "jwt")]
if let Some(sandbox_slot) = SANDBOX_BEARER_SLOT.get()
&& let Err(error) = sandbox_slot.update(
sandbox_refresh.token,
sandbox_refresh.expires_at,
sandbox_refresh.credential_epoch,
)
&& error != crate::jwt::SessionJwtError::StaleCredentialEpoch
{
warn!(%error, "gateway returned an invalid Sandbox Protocol credential");
}
info!("renewed gateway and Sandbox Protocol credentials in-place");
}
Err(status) => {
if status.code() == tonic::Code::Unauthenticated
@@ -464,11 +537,11 @@ async fn refresh_extension_credentials_with_client(
.wrap_err("failed to refresh extension service credentials")?
.into_inner();
// The same refresh response renews the gateway credential. Install it
// before returning so all process-wide gateway clients stay current. This
// is a superset of what the dedicated renewal loop would do, so letting it
// land early is harmless.
install_token_slot(&response.token)?;
let gateway_bearer = validate_gateway_bearer(&response.token)?;
#[cfg(feature = "jwt")]
let sandbox_refresh = validate_sandbox_refresh(&response)
.into_diagnostic()
.wrap_err("gateway returned an invalid Sandbox Protocol credential")?;
// Validate the whole response before mutating any slot, so a malformed or
// partial reply cannot leave the store half-rotated.
@@ -497,6 +570,32 @@ async fn refresh_extension_credentials_with_client(
}
let now_ms = now_ms();
for (token, expires_at_ms) in validated.values() {
BearerTokenSlot::new(token, *expires_at_ms)
.into_diagnostic()
.wrap_err("gateway returned an invalid extension credential")?;
}
// Commit only after the gateway, Sandbox Protocol, and extension
// credentials have all been parsed and validated. The remaining updates
// repeat those validations but cannot fail for the validated inputs.
install_validated_token_slot(gateway_bearer);
#[cfg(feature = "jwt")]
if let Some(sandbox_slot) = SANDBOX_BEARER_SLOT.get() {
match sandbox_slot.update(
sandbox_refresh.token,
sandbox_refresh.expires_at,
sandbox_refresh.credential_epoch,
) {
Ok(()) | Err(crate::jwt::SessionJwtError::StaleCredentialEpoch) => {}
Err(error) => {
return Err(miette::miette!(
"validated Sandbox Protocol credential could not be installed: {error}"
));
}
}
}
let mut selected = HashMap::with_capacity(validated.len());
for (name, (token, expires_at_ms)) in validated {
let slot = store
@@ -556,6 +655,22 @@ fn parse_jwt_exp_ms(jwt: &str) -> Option<i64> {
mod auth_tests {
use super::*;
#[cfg(feature = "jwt")]
#[test]
fn sandbox_refresh_validation_rejects_invalid_lifetime_before_installation() {
let response = crate::proto::RefreshSandboxTokenResponse {
sandbox_token: "sandbox-token".to_string(),
sandbox_expires_at_ms: 999,
credential_epoch: 2,
..Default::default()
};
assert_eq!(
validate_sandbox_refresh(&response).err(),
Some(crate::jwt::SessionJwtError::InvalidLifetime)
);
}
#[test]
fn parse_jwt_exp_reads_unsigned_payload() {
use base64::Engine as _;
crates/openshell-core/src/jwt.rs
+1013 -4
View File
File diff suppressed because it is too large Load Diff
crates/openshell-core/src/lib.rs
+4
View File
@@ -44,7 +44,10 @@ pub mod proposals;
pub mod proto;
pub mod proto_struct;
pub mod provider_credentials;
pub mod rpc_error;
pub mod sandbox_env;
pub mod sandbox_generation;
pub mod sandbox_session;
pub mod secrets;
pub mod settings;
pub mod shell;
@@ -65,6 +68,7 @@ pub use error::{ComputeDriverError, Error, Result};
pub use metadata::{
GetResourceVersion, ObjectId, ObjectLabels, ObjectName, ObjectWorkspace, SetResourceVersion,
};
pub use sandbox_session::{SandboxSessionId, SandboxSessionIdError};
/// Build version string derived from git metadata.
///
crates/openshell-core/src/proposals.rs
+2 -2
View File
@@ -4,7 +4,7 @@
//! Shared state controlling agent-driven policy proposals.
//!
//! Initialised once during sandbox start from the `agent_policy_proposals_enabled`
//! setting and updated by the policy poll loop or authoritative sidecar control
//! setting and updated by the policy poll loop or authoritative supervisor
//! when the setting changes. Read by the `policy.local` route handler and by
//! the skills installer to gate the agent-controlled mutation surface.
@@ -15,7 +15,7 @@ use std::sync::atomic::{AtomicBool, Ordering};
///
/// Clones point at the same atomic value, so the sandbox orchestrator can pass
/// this into the process and network supervisors and then update it from the
/// settings poll loop or sidecar control.
/// settings poll loop or supervisor control.
#[derive(Clone, Debug)]
pub struct AgentProposals {
enabled: Arc<AtomicBool>,
crates/openshell-core/src/provider_credentials.rs
+126 -6
View File
@@ -171,8 +171,8 @@ impl ProviderCredentialState {
/// Build a static provider state from an already-prepared child
/// environment snapshot.
///
/// Kubernetes sidecar topology uses this in the process-only supervisor:
/// the network sidecar owns provider credential resolvers and sends the
/// The Kubernetes sandbox runtime uses this in the sandbox process:
/// the supervisor owns provider credential resolvers and sends the
/// workload-facing env map over a local control channel. The process leaf
/// must inject that map into child processes without re-placeholderizing it
/// or holding the gateway-side resolver material.
@@ -407,12 +407,35 @@ impl ProviderCredentialState {
/// here so SDKs can read them at startup.
/// 3. Everything else stays as placeholders for proxy-time resolution.
pub fn child_env_with_gcp_resolved(&self) -> HashMap<String, String> {
use crate::google_cloud;
let inner = self
.inner
.read()
.expect("provider credential state poisoned");
Self::resolve_child_env_snapshot(&inner).1
}
/// Return the current revision and its workload-facing environment from
/// one state snapshot.
///
/// Remote isolation boundaries use the pair as a revisioned update. The
/// revision must describe the exact environment sent across the boundary,
/// so callers must not obtain the two values through separate lock
/// acquisitions.
pub fn child_env_snapshot_with_gcp_resolved(
&self,
) -> std::io::Result<(u64, HashMap<String, String>)> {
let inner = self
.inner
.read()
.map_err(|_| std::io::Error::other("provider credential state poisoned"))?;
Ok(Self::resolve_child_env_snapshot(&inner))
}
fn resolve_child_env_snapshot(
inner: &ProviderCredentialStateInner,
) -> (u64, HashMap<String, String>) {
use crate::google_cloud;
let mut env = inner.current.child_env.clone();
let has_gcp_metadata = env.contains_key("GCE_METADATA_HOST")
@@ -424,7 +447,7 @@ impl ProviderCredentialState {
.any(|key| env.contains_key(*key) && inner.non_secret_environment_keys.contains(*key));
if !has_gcp_metadata && !has_gcp_config {
return env;
return (inner.current.revision, env);
}
if has_gcp_metadata {
@@ -462,7 +485,44 @@ impl ProviderCredentialState {
}
}
env
(inner.current.revision, env)
}
/// Compare and install a workload-facing environment snapshot.
///
/// Provider environment revisions are opaque content identities, not
/// ordered counters. The expected revision makes retries idempotent while
/// rejecting updates based on a stale view of the boundary state.
pub fn compare_and_install_child_env_snapshot(
&self,
expected_revision: u64,
revision: u64,
mut child_env: HashMap<String, String>,
) -> std::io::Result<u64> {
let mut inner = self
.inner
.write()
.map_err(|_| std::io::Error::other("provider credential state poisoned"))?;
if revision == inner.current.revision || expected_revision != inner.current.revision {
return Ok(inner.current.revision);
}
for key in &inner.suppressed_keys {
child_env.remove(key);
}
inner.current = Arc::new(ProviderCredentialSnapshot {
revision,
child_env,
dynamic_credentials: HashMap::new(),
});
inner.generations.clear();
inner.current_resolver = None;
inner.combined_resolver = None;
inner.non_secret_environment_keys.clear();
inner.static_credential_bindings.clear();
inner.known_static_credential_keys.clear();
inner.static_credential_identity_epochs.clear();
Ok(revision)
}
/// Return the GCP token placeholder and its remaining lifetime in seconds.
@@ -2327,6 +2387,66 @@ mod tests {
);
}
#[test]
fn child_env_snapshot_update_uses_opaque_revision_cas() {
let state = ProviderCredentialState::from_child_env_snapshot(
4,
HashMap::from([("TOKEN".to_string(), "four".to_string())]),
);
assert_eq!(
state
.compare_and_install_child_env_snapshot(
4,
6,
HashMap::from([("TOKEN".to_string(), "six".to_string())]),
)
.unwrap(),
6
);
assert_eq!(
state
.compare_and_install_child_env_snapshot(
4,
5,
HashMap::from([("TOKEN".to_string(), "stale".to_string())]),
)
.unwrap(),
6
);
assert_eq!(
state
.compare_and_install_child_env_snapshot(6, 2, HashMap::new())
.unwrap(),
2,
"opaque revisions may move numerically backwards"
);
let (revision, env) = state.child_env_snapshot_with_gcp_resolved().unwrap();
assert_eq!(revision, 2);
assert!(env.is_empty(), "an empty snapshot must revoke the old env");
}
#[test]
fn poisoned_environment_update_returns_error_without_recovering_state() {
let state = ProviderCredentialState::from_child_env_snapshot(4, HashMap::new());
let poison = state.clone();
assert!(
std::thread::spawn(move || {
let _guard = poison.inner.write().unwrap();
panic!("poison state during mutation");
})
.join()
.is_err()
);
assert!(
state
.compare_and_install_child_env_snapshot(4, 5, HashMap::new())
.is_err()
);
assert!(state.child_env_snapshot_with_gcp_resolved().is_err());
}
#[test]
fn stale_generation_falls_back_to_current_credential_after_retention_window() {
let state = ProviderCredentialState::from_environment(
crates/openshell-core/src/rpc_error.rs
+172
View File
@@ -0,0 +1,172 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
//! Standard gRPC error details shared by gateway handlers and SDK clients.
//!
//! Retry guidance describes recovery from a failure. It never establishes that
//! repeating an arbitrary mutation is safe.
use std::collections::HashMap;
use std::time::Duration;
use prost::Message;
use tonic::{Code, Status};
pub use tonic_types::{ErrorDetails, StatusExt};
/// Domain of gateway-owned machine-readable error reasons.
pub const ERROR_DOMAIN: &str = "openshell.nvidia.com";
/// Decode supported details without trusting inconsistent envelopes or delays.
/// Unknown and malformed details remain available on the original status.
pub fn decode_details(status: &Status) -> Option<ErrorDetails> {
let envelope = tonic_types::pb::Status::decode(status.details()).ok()?;
if envelope.code != status.code() as i32 || envelope.message != status.message() {
return None;
}
let mut details = ErrorDetails::new();
for detail in envelope.details {
match detail.type_url.rsplit('/').next() {
Some("google.rpc.BadRequest") => {
if let Ok(value) = tonic_types::pb::BadRequest::decode(detail.value.as_slice()) {
for violation in value.field_violations {
details.add_bad_request_violation(violation.field, violation.description);
}
}
}
Some("google.rpc.ErrorInfo") => {
if let Ok(value) = tonic_types::pb::ErrorInfo::decode(detail.value.as_slice()) {
details.set_error_info(value.reason, value.domain, value.metadata);
}
}
Some("google.rpc.RetryInfo") => {
if let Ok(value) = tonic_types::pb::RetryInfo::decode(detail.value.as_slice())
&& let Some(delay) = value.retry_delay
&& (0..=315_576_000_000).contains(&delay.seconds)
&& (0..1_000_000_000).contains(&delay.nanos)
{
details.set_retry_info(Some(Duration::new(
delay.seconds.cast_unsigned(),
delay.nanos.cast_unsigned(),
)));
}
}
_ => {}
}
}
Some(details)
}
/// Return a field-level validation failure without reflecting the field value.
pub fn invalid_argument(field: &str, message: impl Into<String>) -> Status {
let message = message.into();
let mut details = ErrorDetails::with_bad_request_violation(field, message.clone());
details.set_error_info("INVALID_ARGUMENT", ERROR_DOMAIN, HashMap::new());
Status::with_error_details(Code::InvalidArgument, message, details)
}
/// Return a precondition failure with a stable reason.
pub fn failed_precondition(reason: &str, message: impl Into<String>) -> Status {
Status::with_error_details(
Code::FailedPrecondition,
message,
ErrorDetails::with_error_info(reason, ERROR_DOMAIN, HashMap::new()),
)
}
/// A conditional write lost a race. Read fresh state before trying a new write.
pub fn resource_version_conflict(message: impl Into<String>, version: Option<u64>) -> Status {
let mut metadata = HashMap::from([("recovery".into(), "REFRESH_STATE".into())]);
if let Some(version) = version {
metadata.insert("current_resource_version".into(), version.to_string());
}
Status::with_error_details(
Code::Aborted,
message,
ErrorDetails::with_error_info("RESOURCE_VERSION_CONFLICT", ERROR_DOMAIN, metadata),
)
}
/// Return a transient failure with a minimum delay for retry-safe operations.
pub fn unavailable(reason: &str, message: impl Into<String>, delay: Duration) -> Status {
let mut details = ErrorDetails::with_error_info(reason, ERROR_DOMAIN, HashMap::new());
details.set_retry_info(Some(delay));
Status::with_error_details(Code::Unavailable, message, details)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn decode_ignores_invalid_delay_and_preserves_other_details() {
let status = invalid_argument("name", "invalid name");
let mut envelope = tonic_types::pb::Status::decode(status.details()).unwrap();
envelope.details.push(prost_types::Any {
type_url: "type.googleapis.com/google.rpc.RetryInfo".into(),
value: tonic_types::pb::RetryInfo {
retry_delay: Some(prost_types::Duration {
seconds: -1,
nanos: 0,
}),
}
.encode_to_vec(),
});
envelope.details.push(prost_types::Any {
type_url: "type.googleapis.com/google.rpc.BadRequest".into(),
value: vec![255],
});
let status = Status::with_details(
status.code(),
status.message(),
envelope.encode_to_vec().into(),
);
let details = decode_details(&status).unwrap();
assert!(details.retry_info().is_none());
assert_eq!(
details.bad_request().unwrap().field_violations[0].field,
"name"
);
let mismatched = Status::with_details(
Code::Internal,
status.message(),
status.details().to_vec().into(),
);
assert!(decode_details(&mismatched).is_none());
}
#[test]
fn validation_identifies_the_field_and_reason() {
let status = invalid_argument("spec.command[0]", "must not be empty");
let details = status.get_error_details();
assert_eq!(status.code(), Code::InvalidArgument);
let violation = &details.bad_request().unwrap().field_violations[0];
assert_eq!(violation.field, "spec.command[0]");
assert_eq!(violation.description, "must not be empty");
assert_eq!(details.error_info().unwrap().domain, ERROR_DOMAIN);
assert!(details.retry_info().is_none());
}
#[test]
fn conflict_requires_fresh_state_without_blind_retry_delay() {
let status = resource_version_conflict("concurrent write", Some(17));
let details = status.get_error_details();
let info = details.error_info().unwrap();
assert_eq!(status.code(), Code::Aborted);
assert_eq!(info.reason, "RESOURCE_VERSION_CONFLICT");
assert_eq!(info.metadata["recovery"], "REFRESH_STATE");
assert_eq!(info.metadata["current_resource_version"], "17");
assert!(details.retry_info().is_none());
}
#[test]
fn temporary_unavailability_has_a_reason_and_delay() {
let status = unavailable("GATEWAY_NOT_READY", "try later", Duration::from_secs(1));
let details = status.get_error_details();
assert_eq!(status.code(), Code::Unavailable);
assert_eq!(details.error_info().unwrap().reason, "GATEWAY_NOT_READY");
assert_eq!(
details.retry_info().unwrap().retry_delay,
Some(Duration::from_secs(1))
);
}
}
crates/openshell-core/src/sandbox_env.rs
+24 -5
View File
@@ -39,7 +39,8 @@ const MAIN_PROCESS_SPEC_BASE64URL_PREFIX: &str = "base64url:";
pub struct MainProcessConfig {
pub version: u32,
/// Canonical command. Empty means "no command supplied": the supervisor
/// resolves the default login shell against the sandbox image. A non-empty
/// asks the sandbox boundary to resolve the default login shell against
/// the agent image. A non-empty
/// command is the exact program+args and is run verbatim.
pub command: Vec<String>,
pub tty: bool,
@@ -51,8 +52,8 @@ impl MainProcessConfig {
pub const VERSION: u32 = 1;
/// Default config for a sandbox created without a command. The command is
/// left empty on purpose: the supervisor picks a login shell that exists in
/// the sandbox image (bash when present, otherwise `/bin/sh`). A TTY is
/// left empty on purpose: the sandbox boundary picks a login shell that
/// exists in the agent image (bash when present, otherwise `/bin/sh`). A TTY is
/// requested because the default is an interactive login shell.
#[must_use]
pub fn scratch() -> Self {
@@ -99,7 +100,7 @@ impl MainProcessConfig {
));
}
// An empty command is valid: it means "no command supplied", and the
// supervisor resolves the default login shell. Only a present-but-blank
// sandbox boundary resolves the default login shell. Only a present-but-blank
// program is rejected.
if !config.command.is_empty() && config.command[0].is_empty() {
return Err(format!(
@@ -134,6 +135,13 @@ pub const TELEMETRY_ENABLED: &str = "OPENSHELL_TELEMETRY_ENABLED";
/// `"sidecar"`; the default combined supervisor path omits it.
pub const SUPERVISOR_TOPOLOGY: &str = "OPENSHELL_SUPERVISOR_TOPOLOGY";
/// The isolation backend admitted by the deployment configuration (RFC 0012).
///
/// Delivered on a channel separate from the topology descriptor so descriptor
/// verification against the admitted backend is not self-referential. Required
/// whenever a topology descriptor is supplied.
pub const ADMITTED_ISOLATION_BACKEND: &str = "OPENSHELL_ADMITTED_ISOLATION_BACKEND";
/// Network enforcement backend selected by the compute driver.
pub const NETWORK_ENFORCEMENT_MODE: &str = "OPENSHELL_NETWORK_ENFORCEMENT_MODE";
@@ -166,6 +174,17 @@ pub const GATEWAY_TLS_SERVER_NAME: &str = "OPENSHELL_GATEWAY_TLS_SERVER_NAME";
/// by workload child processes.
pub const PROXY_TLS_DIR: &str = "OPENSHELL_PROXY_TLS_DIR";
/// Optional path to a durable PEM-encoded interception CA certificate.
/// Must be configured together with [`PROXY_CA_KEY`].
pub const PROXY_CA_CERT: &str = "OPENSHELL_PROXY_CA_CERT";
/// Optional path to the private key for [`PROXY_CA_CERT`].
/// Must be configured together with the certificate path.
pub const PROXY_CA_KEY: &str = "OPENSHELL_PROXY_CA_KEY";
/// Whether the control-owned SSH Unix socket is shared across trusted UIDs.
pub const SSH_SOCKET_SHARED: &str = "OPENSHELL_SSH_SOCKET_SHARED";
/// Path to the CA certificate for mTLS communication with the gateway.
pub const TLS_CA: &str = "OPENSHELL_TLS_CA";
@@ -298,7 +317,7 @@ mod tests {
#[test]
fn omitted_command_stays_empty_for_supervisor_resolution() {
// No command supplied → empty command; the supervisor resolves the
// No command supplied → empty command; the sandbox boundary resolves the
// default login shell against the sandbox image.
let empty = crate::proto::compute::v1::DriverSandboxSpec::default();
assert!(
crates/openshell-core/src/sandbox_generation.rs
+95
View File
@@ -0,0 +1,95 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
use std::fmt;
use serde::{Deserialize, Serialize};
/// Identifies one requested start of a stable sandbox.
///
/// The gateway derives this value from the durable lifecycle transition so a
/// retried driver call carries the same identity after process restart.
#[derive(Clone, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)]
#[serde(transparent)]
pub struct SandboxGenerationId(String);
#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)]
pub enum SandboxGenerationIdError {
#[error("sandbox generation ID is required")]
Empty,
#[error("sandbox generation ID exceeds 64 characters")]
TooLong,
#[error("sandbox generation ID contains an unsupported character")]
InvalidCharacter,
}
impl SandboxGenerationId {
pub const MAX_LEN: usize = 64;
/// Construct the stable generation assigned to a gateway start
/// transition. Resource versions are monotonic within one sandbox record.
#[must_use]
pub fn from_start_resource_version(resource_version: u64) -> Self {
Self(format!("g{resource_version:016x}"))
}
pub fn parse(value: impl Into<String>) -> Result<Self, SandboxGenerationIdError> {
let value = value.into();
if value.is_empty() {
return Err(SandboxGenerationIdError::Empty);
}
if value.len() > Self::MAX_LEN {
return Err(SandboxGenerationIdError::TooLong);
}
if !value
.bytes()
.all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'-')
{
return Err(SandboxGenerationIdError::InvalidCharacter);
}
Ok(Self(value))
}
#[must_use]
pub fn as_str(&self) -> &str {
&self.0
}
#[must_use]
pub fn into_string(self) -> String {
self.0
}
}
impl fmt::Display for SandboxGenerationId {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter.write_str(self.as_str())
}
}
#[cfg(test)]
mod tests {
use crate::sandbox_generation::{SandboxGenerationId, SandboxGenerationIdError};
#[test]
fn resource_version_generation_is_stable_and_valid() {
let generation = SandboxGenerationId::from_start_resource_version(42);
assert_eq!(generation.as_str(), "g000000000000002a");
assert_eq!(
SandboxGenerationId::parse(generation.to_string()),
Ok(generation)
);
}
#[test]
fn rejects_values_that_are_not_safe_runtime_identifiers() {
assert_eq!(
SandboxGenerationId::parse(""),
Err(SandboxGenerationIdError::Empty)
);
assert_eq!(
SandboxGenerationId::parse("Generation/1"),
Err(SandboxGenerationIdError::InvalidCharacter)
);
}
}
crates/openshell-core/src/sandbox_session.rs
+119
View File
@@ -0,0 +1,119 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
//! Strong identity for one sandbox runtime launch.
use std::{fmt, str::FromStr};
use serde::{Deserialize, Deserializer, Serialize, Serializer, de};
use thiserror::Error;
use uuid::Uuid;
/// Identifies one create or start-from-stopped sandbox launch.
///
/// Retries of the same durable launch reuse this value. A later launch gets a
/// new value even when the compute platform reuses the same sandbox resource.
#[derive(Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct SandboxSessionId(Uuid);
impl SandboxSessionId {
/// Generate a fresh launch identity.
#[must_use]
pub fn new() -> Self {
Self(Uuid::new_v4())
}
/// Return the underlying UUID.
#[must_use]
pub const fn as_uuid(&self) -> &Uuid {
&self.0
}
}
impl Default for SandboxSessionId {
fn default() -> Self {
Self::new()
}
}
impl fmt::Display for SandboxSessionId {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
self.0.hyphenated().fmt(formatter)
}
}
impl fmt::Debug for SandboxSessionId {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_tuple("SandboxSessionId")
.field(&self.to_string())
.finish()
}
}
impl FromStr for SandboxSessionId {
type Err = SandboxSessionIdError;
fn from_str(value: &str) -> Result<Self, Self::Err> {
let parsed = Uuid::parse_str(value).map_err(|_| SandboxSessionIdError)?;
if parsed.is_nil() || parsed.hyphenated().to_string() != value {
return Err(SandboxSessionIdError);
}
Ok(Self(parsed))
}
}
impl Serialize for SandboxSessionId {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
serializer.collect_str(self)
}
}
impl<'de> Deserialize<'de> for SandboxSessionId {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: Deserializer<'de>,
{
let value = String::deserialize(deserializer)?;
value.parse().map_err(|_| {
de::Error::invalid_value(
de::Unexpected::Str(&value),
&"a non-nil canonical lowercase UUID",
)
})
}
}
/// A sandbox session ID was not a non-nil canonical lowercase UUID.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error)]
#[error("sandbox session ID must be a non-nil canonical lowercase UUID")]
pub struct SandboxSessionIdError;
#[cfg(test)]
mod tests {
use crate::sandbox_session::SandboxSessionId;
#[test]
fn round_trips_canonical_id() {
let session = SandboxSessionId::new();
let encoded = serde_json::to_string(&session).expect("serialize session ID");
let decoded: SandboxSessionId =
serde_json::from_str(&encoded).expect("deserialize session ID");
assert_eq!(decoded, session);
}
#[test]
fn rejects_nil_and_noncanonical_ids() {
for invalid in [
"00000000-0000-0000-0000-000000000000",
"550E8400-E29B-41D4-A716-446655440000",
"550e8400e29b41d4a716446655440000",
"not-a-uuid",
] {
assert!(invalid.parse::<SandboxSessionId>().is_err(), "{invalid}");
}
}
}
crates/openshell-core/src/shell.rs
+4 -3
View File
@@ -3,7 +3,7 @@
//! Login-shell resolution for sandbox images.
//!
//! The default sandbox command and the interactive SSH session need a shell,
//! The default sandbox command and interactive SSH sessions need a shell,
//! but not every base image ships the same one. Debian-based images provide
//! `bash`; minimal images such as Alpine only provide `/bin/sh` (`BusyBox`
//! `ash`). Hard-coding `/bin/bash` makes sandbox startup fail on those images
@@ -51,8 +51,9 @@ pub fn is_executable(path: &str) -> bool {
/// Resolve a login shell that exists in the current root filesystem.
///
/// Tries [`SHELL_CANDIDATES`] in order and falls back to [`POSIX_SH`]. Because
/// this inspects the filesystem, call it from the supervisor (inside the
/// sandbox), never on the gateway.
/// this inspects the filesystem, call it from the sandbox boundary or another
/// process inside the workload filesystem, never from the external supervisor
/// or gateway.
///
/// `$SHELL` is intentionally not consulted: it is image/user-controlled, the
/// result is later invoked with `-lc`, and an executable that is not a
crates/openshell-driver-docker/Cargo.toml
+9 -3
View File
@@ -16,6 +16,8 @@ path = "src/main.rs"
[dependencies]
openshell-core = { path = "../openshell-core", default-features = false, features = ["driver-extraction"] }
openshell-isolation-interface = { path = "../openshell-isolation-interface" }
openshell-sandbox-backend = { path = "../openshell-sandbox-backend" }
openshell-otel = { path = "../openshell-otel" }
opentelemetry = { workspace = true }
@@ -38,15 +40,19 @@ miette = { workspace = true }
toml = { workspace = true }
tower-http = { workspace = true }
http = { workspace = true }
rand = { workspace = true }
sha2 = { workspace = true }
rustix = { workspace = true }
libc = "0.2"
tar = "0.4"
tempfile = "3"
[dev-dependencies]
openshell-otel-test-support = { path = "../openshell-otel-test-support" }
opentelemetry = { workspace = true }
opentelemetry_sdk = { workspace = true, features = ["testing"] }
prost-types = { workspace = true }
tar = "0.4"
temp-env = { version = "0.3", features = ["async_closure"] }
tempfile = "3"
temp-env = "0.3"
tracing-subscriber = { workspace = true }
[lints]
crates/openshell-driver-docker/README.md
+110 -171
View File
@@ -1,148 +1,115 @@
# openshell-driver-docker
Docker-backed compute driver for local OpenShell gateways.
Docker-backed compute driver for local and remote OpenShell gateways.
When the gateway configures `[openshell.gateway.otlp]`, Docker compute-driver
spans export to the same OTLP/gRPC collector with the service name
`openshell-driver-docker`. The in-process driver preserves the gateway trace
context and emits the compute-driver RPC boundary that a standalone driver
would expose.
The driver uses `bollard` to manage sandbox resources through the configured
Docker API socket. When `socket_path` is unset, it selects the first standard
local socket that responds to an API ping. An explicitly selected Docker driver
falls back to `/var/run/docker.sock` when no candidate responds.
`mise run gateway:docker` enables this export only when a local collector is
listening on `127.0.0.1:4317`. Otherwise, it omits the gateway OTLP configuration
so the development gateway does not repeatedly report export failures.
The standalone `openshell-driver-docker` binary accepts
`OPENSHELL_OTLP_ENDPOINT`. When set, it exports Docker driver spans to that
collector, continues W3C trace context from gateway RPC metadata, and flushes
spans during graceful shutdown.
The driver manages sandbox containers through the local Docker daemon with the
`bollard` client. It is intended for developer environments where Docker is
already available and running Kubernetes would be unnecessary.
The driver connects to `[openshell.drivers.docker].socket_path` when configured.
Otherwise, it uses the first standard local Docker socket that responds to an
API ping, which is the same selection mechanism used by gateway auto-detection.
An explicitly selected Docker driver falls back to `/var/run/docker.sock` when
no candidate responds.
When the gateway configures `[openshell.gateway.otlp]`, the in-process driver
exports spans to the same OTLP/gRPC collector as
`openshell-driver-docker`. The standalone driver accepts
`OPENSHELL_OTLP_ENDPOINT`, continues W3C trace context from gateway RPC
metadata, and flushes spans during graceful shutdown.
## Runtime Model
The gateway runs as a host process. The Docker driver creates one container per
sandbox and starts the `openshell-sandbox` supervisor inside that container. The
supervisor then creates the nested sandbox namespace for the agent process.
The driver creates two containers for each sandbox:
## Stop and Start
- `openshell-sandbox` is PID 1 in the workload container. It owns the workload
process tree, seccomp notification broker, mandatory Landlock baseline,
binary identity, exec/signal/wait/PTY operations, and loopback forwarding.
- `openshell-supervisor` runs in a separate companion container. It owns the
gateway session, policy engine, credentials, interception CA, SSH relay, L7
inspection, DNS policy, and external upstream connections.
Stop stops the managed container without removing it. Docker retains the
container writable layer, attached volumes, labels, token material, and restart
policy. Start starts that same container, so files in the resolved OCI
workspace remain available. A durably stopped sandbox is excluded from
gateway startup recovery and stays stopped across gateway restarts. Delete
continues to force-remove the container and clean up driver-owned material.
Graceful gateway shutdown sends `StopSandbox` for each sandbox whose persisted
phase requires running compute without changing that persisted intent. On
startup, the gateway sends an idempotent `StartSandbox` request for the same
sandboxes, restarting their retained containers. Explicitly stopped sandboxes
remain excluded.
Both containers are non-root, request no capabilities, and set
no-new-privileges. A shared named volume carries the authenticated Unix socket
and sandbox bootstrap material. A second supervisor-only volume carries the
supervisor JWT and gateway client credentials and is never mounted into the
workload.
Before creating the container, the driver inspects the final sandbox image and
captures its immutable image ID, raw OCI `Config.User`, and OCI
`Config.WorkingDir`. Container creation uses that image ID, preventing a
mutable tag from changing between inspection and launch. The supervisor runs as
root, resolves omitted policy identity fields from the image declaration, and
drops only agent children to the resulting identity. Named OCI components
remain names after validation; a missing group is filled with the user's
numeric primary GID. Explicit `process.run_as_user` and
`process.run_as_group` values take precedence independently.
The workload uses `network_mode=none`. Its seccomp user-notification broker
mediates every supported TCP and DNS operation, attributes it to the calling
binary, and sends the request across the private channel. The supervisor
authorizes the request before it opens an upstream connection. Docker's absent
workload network is the mandatory outer fence if mediation fails or is
bypassed. Only the trusted supervisor companion joins the driver-owned bridge,
where it originates approved egress and can resolve other services on that
network.
An absolute OCI working directory becomes the agent workspace. An empty,
root (`/`), or explicit `/sandbox` declaration uses `/sandbox`, which OpenShell
creates when necessary and owns as a compatibility workspace. Any other image
workdir must already exist without symlink components. The completed identity,
including supplementary groups, must already be able to traverse every parent
and write and enter the workdir. OpenShell does not change its ownership or
mode.
The driver copies trusted runtime bytes from the configured supervisor image
through the Docker archive API. No workload launch depends on a host bind
mount or a tool supplied by the workload image, so the same path works with
local, remote, and VM-backed Docker daemons.
OpenShell deliberately asks the Linux kernel to make this access decision
under the completed sandbox identity instead of reproducing permission rules
from ownership and mode bits. Mode-bit inspection alone can reject authority
granted by a POSIX ACL or overlook a denial imposed by a Linux Security Module
such as SELinux or AppArmor. OpenShell does not configure or otherwise manage
ACLs or LSM policy here; the one-shot validator only observes the kernel's
effective decision. This keeps the no-authority-expansion invariant aligned
with the access the eventual workload will receive without adding a separate,
incomplete permission model to OpenShell.
## Identity and Workspace
Image `VOLUME` declarations must not cover the workdir or one of its parents
because Docker would mount the volume before the supervisor could validate the
immutable image path.
Workdirs under the standard OCI runtime namespaces `/proc`, `/sys`, and `/dev`
are rejected, as are paths that overlap concrete OpenShell control resources.
The workspace is the child cwd and `HOME`. The supervisor starts from `/`, then
reports an invalid workdir as a readiness failure.
Before creating the workload, the driver pins the image ID and reads its
passwd/group databases through a stopped metadata container. It resolves the
admitted policy identity, or the image `Config.User` fallback, into one exact
non-root UID, primary GID, and supplementary-group set. Docker launches
`openshell-sandbox` with that identity, and the sandbox uses the same identity
for every canonical and exec process. UID or GID zero and unresolved symbolic
identities are rejected.
Docker containers join an OpenShell-managed bridge network. The driver injects
`host.openshell.internal` and `host.docker.internal` so supervisors have stable
names for reaching the gateway host. On Docker Desktop, Colima, Rancher
Desktop, OrbStack, and macOS-hosted gateways, those names use Docker's
`host-gateway` alias. The driver requests a separate IPv4 loopback callback
listener when the primary listener does not already cover it. On native Linux
Docker, the gateway also binds the bridge gateway IP so containers can call
back to the host process.
An absolute OCI working directory becomes the workspace. An empty, root (`/`),
or explicit `/sandbox` declaration uses `/sandbox`. Any other workdir must
already exist without symlink components. The resolved identity must be able to
traverse every parent and write and enter the workdir; OpenShell does not
change its ownership or mode.
Image `VOLUME` declarations and user mounts must not cover the workdir, one of
its parents, or the reserved `/.openshell` runtime/channel tree. OpenShell asks
the kernel to validate access under the final identity, so POSIX ACL and host
LSM decisions remain authoritative.
## Container Contract
The driver-controlled container settings are part of the sandbox security
contract:
| Setting | Purpose |
|---|---|
| `user = "0"` | The supervisor needs root inside the container to prepare namespaces, mounts, Landlock, and seccomp. |
| `network_mode = openshell` | Places the supervisor on the managed Docker bridge network. |
| `cap_add` | Grants supervisor-only capabilities required for namespace setup and process inspection. |
| `apparmor=unconfined` | Avoids Docker's default profile blocking required mount operations. |
| `restart_policy = no` | A canonical main-process exit remains terminal and is not silently restarted by Docker. |
| `PidsLimit` | Enforces the sandbox PID budget at the Docker cgroup layer. `[openshell.drivers.docker].sandbox_pids_limit` defaults to `2048`; explicit `0` is invalid. |
| CDI GPU request | Uses opaque `driver_config.cdi_devices` values when set; otherwise selects the requested count of NVIDIA CDI GPUs in round-robin order when daemon CDI support is detected. Docker daemon `/info` can permit `nvidia.com/gpu=all` as a WSL2 all-only compatibility fallback, where it counts as one selectable device. Exact CDI device lists must not contain duplicates and must match the effective GPU count. |
| `policy-dns-transparent-tcp` capability | Declares that the combined Docker supervisor can own namespace-local DNS/TCP capture and coupled workload restart. The shared supervisor still owns DNS eligibility, mappings, authorization, pinned dialing, relaying, and OCSF decisions. The marker is stripped from the workload environment. |
| Exact non-root `user` and `group_add` | Gives sandbox and workload the same immutable UID/GID/group identity required for capability-free observation. |
| `cap_drop = ALL`, no `cap_add`, no-new-privileges | Prevents either container from acquiring Linux capabilities. |
| Docker default seccomp and AppArmor profiles | Retains runtime hardening; startup confirmation fails closed if nested seccomp notification is unavailable. |
| `network_mode = none` on the workload | Removes direct external routes. |
| Driver-owned bridge on the supervisor | Lets the trusted supervisor originate approved gateway and upstream connections and use Docker service discovery. |
| `restart_policy = no` | Keeps canonical main-process exit terminal. |
| `PidsLimit` | Applies the configured sandbox PID budget. Omit `sandbox_pids_limit` to use OpenShell's default. Explicit zero is invalid. |
| Private named volumes | One carries the authenticated sandbox/supervisor channel. The other is mounted only into the supervisor and contains its JWT and private gateway credentials. |
| In-memory `/run/openshell-supervisor-ca` tmpfs | Holds only the public supervisor CA certificate and trust bundle without making all of `/run` writable. |
| CDI GPU request | Assigns the exact validated CDI devices requested by driver config or count-based selection. |
The agent child process does not retain these supervisor privileges.
## Stop, Start, and Delete
Stop terminates the supervisor companion and stops the workload container
without removing it. Docker retains the workload writable layer and attached
volumes. Start stages a fresh sandbox bootstrap bundle, restarts that workload,
and creates a new supervisor companion. A durably stopped sandbox stays stopped
across gateway restarts.
Delete force-removes both containers, the driver-owned runtime volumes, and the
host-private runtime descriptor. Missing or altered descriptor and channel resources
fail closed; the driver does not run an older combined-supervisor layout.
## Driver Config Mounts
The gateway forwards the `docker` block from `--driver-config-json` to this
driver. The driver accepts user-supplied `mounts` entries with these Docker
mount types:
The gateway forwards the `docker` block from `--driver-config-json`. Supported
mount types are:
- `bind`: mounts an absolute host path when `[openshell.drivers.docker]`
has `enable_bind_mounts = true`.
- `volume`: mounts an existing Docker named volume. The driver validates that
the volume exists before provisioning and never creates or removes it.
Docker local-driver volumes created with bind options are treated as host
bind mounts and require `enable_bind_mounts = true`.
- `tmpfs`: mounts an in-memory filesystem with optional `options`,
`size_bytes`, and `mode`.
- `bind`: an absolute daemon-host path, allowed only when
`[openshell.drivers.docker].enable_bind_mounts = true`.
- `volume`: an existing named volume. The driver never creates or removes a
user-supplied volume. Bind-backed local volumes require
`enable_bind_mounts = true`.
- `tmpfs`: an in-memory filesystem with optional size and mode.
Host bind mounts are disabled by default because they expose gateway host
paths to sandbox requests. Image mounts are not part of the Docker
driver-config schema. The driver still uses internal bind mounts for
OpenShell-owned supervisor, token, and TLS material.
Host bind mounts are disabled by default because they expose daemon-host paths
to sandbox requests. User bind and volume mounts are read-only by default.
Targets must be absolute, normalized paths and cannot overlap the workspace
root or OpenShell control paths.
Docker `bind` mounts accept `source`, `target`, optional `read_only`, and an
optional `selinux_label` of `shared` (applies `:z`) or `private` (applies
`:Z`) for SELinux-enforcing hosts. Docker `volume` mounts may include
`subpath`. User-supplied bind and volume mounts are read-only by default; set
`read_only: false` to make them writable. Mount `source`, `target`, and
`subpath` values must not contain surrounding whitespace. Mount targets must be
absolute container paths and must not replace or contain the resolved workspace
root. Nested workspace mounts remain valid. Mounts also must not overlap the
configured SSH socket or the reserved `/opt/openshell`, `/etc/openshell`,
`/etc/openshell-tls`, `/run/openshell`, `/run/openshell-sidecar`, and network
namespace roots.
Example named-volume usage:
Example:
```shell
docker volume create openshell-work
@@ -152,70 +119,42 @@ openshell sandbox create \
-- claude
```
## Supervisor Binary Resolution
## Runtime Image
The Docker driver bind-mounts a host-side Linux `openshell-sandbox` binary into
each sandbox container. Resolution order is:
1. `supervisor_bin` in `[openshell.drivers.docker]`.
2. `supervisor_image` in `[openshell.drivers.docker]`, extracting
`/openshell-sandbox` from that image.
3. A sibling `openshell-sandbox` next to the running `openshell-gateway` binary.
4. A local Linux cargo target build for the Docker daemon architecture.
5. The release-matched default supervisor image, extracting `/openshell-sandbox`.
Release and Docker-image gateway builds bake the matching supervisor image tag
into the binary at compile time. The default Docker supervisor image is not
`:latest` unless a custom build explicitly sets that tag.
`sandbox_runtime_image` contains the statically linked musl
`/openshell-sandbox` binary. The driver extracts that binary as bytes and
stages it into the stopped workload. `supervisor_image` contains the
dynamically linked glibc `/openshell-supervisor` binary that runs in the
host-networked supervisor container. Release and gateway image builds bake
matching image tags into the binary.
## Callback and TLS
`OPENSHELL_ENDPOINT` is injected from the gateway's configured gRPC endpoint.
When no endpoint is configured, the driver uses
`host.openshell.internal:<gateway-port>` with the appropriate HTTP or HTTPS
scheme. Set `host_gateway_ip` only when the host has an explicit, locally
assigned address that containers should use for callbacks; package-managed
macOS gateways should leave it unset.
`OPENSHELL_ENDPOINT` and gateway authentication material are injected only into
the supervisor companion. The workload never receives the sandbox JWT, gateway
client TLS key, policy authority, or interception CA private key.
For HTTPS endpoints, the server certificate must include the endpoint host as a
subject alternative name. Docker sandboxes also need the client TLS bundle
mounted into the container and exposed with:
When no endpoint is configured, the driver derives
`host.openshell.internal:<gateway-port>`. Native Linux uses the managed bridge
gateway. Docker Desktop and compatible VM-backed daemons use Docker's
`host-gateway` route. A configured HTTPS server certificate must include the
endpoint host in its subject alternative names.
- `OPENSHELL_TLS_CA`
- `OPENSHELL_TLS_CERT`
- `OPENSHELL_TLS_KEY`
The driver pins a concrete managed-bridge address in the sandbox descriptor.
For Docker's special `host-gateway` route, it leaves the address unpinned so
the supervisor resolves the driver-injected alias in its own container. This
avoids treating the Docker VM's loopback as the desktop host.
HTTP endpoints reject TLS material because the supervisor would not use it.
## Corporate proxy, SPIFFE, and AppArmor
`https_proxy`, `no_proxy`, and `proxy_auth_file` in
`[openshell.drivers.docker]` are operator-owned supervisor settings. Docker
passes the proxy URL and bypass list on the supervisor command line and mounts
an optional `user:pass` auth file read-only at a root-only path. Credentials
never appear in container environment or Docker labels. An auth file used with
an `http://` proxy requires `proxy_auth_allow_insecure = true`; an `https://`
proxy protects the Basic-auth header in its TLS session.
Set `provider_spiffe_workload_api_socket` to an absolute host UNIX socket to
project its dedicated parent directory into the supervisor and set
`OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET` to the guest path. TCP URIs are
rejected for this projection. `app_armor_profile` uses the shared
`RuntimeDefault`, `Unconfined`, or `Localhost/<profile>` vocabulary. Docker
uses explicit `Unconfined` by default because the supervisor's namespace mount
setup is incompatible with `docker-default`; requested confined profiles fail
at startup if Docker does not report AppArmor support.
## Environment Ownership
The driver merges template environment and sandbox spec environment first, then
overwrites security-critical keys:
The supervisor owns these security-critical variables:
- `OPENSHELL_ENDPOINT`
- `OPENSHELL_SANDBOX_ID`
- `OPENSHELL_SANDBOX`
- `OPENSHELL_SANDBOX_TOKEN_FILE`
- `OPENSHELL_SSH_SOCKET_PATH`
- `OPENSHELL_MAIN_PROCESS_SPEC`
- TLS path variables when HTTPS is enabled
Do not allow sandbox images or templates to override these values.
Template and sandbox environment is encoded in the protected bootstrap and
exposed only to workload children. Workload input cannot override
security-critical supervisor variables.
crates/openshell-driver-docker/src/isolation.rs
+172
View File
@@ -0,0 +1,172 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
//! Docker provisioning for the shared authenticated boundary protocol.
//!
//! Docker owns only container placement, the protected socket, and immutable OCI resource
//! claims. Lifecycle, process, network, identity, and wire behavior live in
//! `openshell-isolation-interface` and `openshell-sandbox`.
use std::collections::{BTreeMap, HashMap};
use std::net::IpAddr;
use std::path::PathBuf;
use openshell_isolation_interface::contract::{DriverFenceEvidence, ResolvedWorkloadIdentity};
use openshell_sandbox_backend::GPU_RESOURCE_CLAIM;
use openshell_sandbox_backend::boundary_protocol::{
BoundaryConfig, BoundaryListener, GatewayVerificationKey, SandboxRuntimeDescriptor,
SandboxTlsClientConfig, SandboxTlsServerConfig, SandboxTransport,
};
/// Driver-owned inputs that bind one Docker container to one boundary.
pub struct DockerBoundarySpec {
pub boundary_id: String,
pub generation: String,
pub session_id: openshell_core::SandboxSessionId,
pub session_rotation: openshell_core::jwt::SessionRotation,
pub auth_epoch: openshell_core::jwt::CredentialEpoch,
pub gateway_id: String,
pub verification_keys: Vec<GatewayVerificationKey>,
pub container_id: String,
pub image_identity: String,
pub gpu_requested: bool,
pub listener_socket: PathBuf,
pub control_socket: PathBuf,
pub sandbox_tls: SandboxTlsServerConfig,
pub supervisor_tls: SandboxTlsClientConfig,
pub host_gateway_ip: Option<IpAddr>,
pub workload_identity: ResolvedWorkloadIdentity,
pub child_env: HashMap<String, String>,
}
/// Protected container config and matching host descriptor.
pub struct DockerBoundaryProvisioning {
pub boundary_config: BoundaryConfig,
pub runtime_descriptor: SandboxRuntimeDescriptor,
}
impl DockerBoundarySpec {
/// Produce both sides of the common protocol from the same immutable
/// Docker coordinates so attach cannot bind a different container.
#[must_use]
pub fn provision(self) -> DockerBoundaryProvisioning {
let mut resource_claims = BTreeMap::from([
("docker.container_id".to_string(), self.container_id),
("docker.image_identity".to_string(), self.image_identity),
]);
if self.gpu_requested {
resource_claims.insert(GPU_RESOURCE_CLAIM.to_string(), "true".to_string());
}
let driver_fence = DriverFenceEvidence::Docker {
container_id: resource_claims["docker.container_id"].clone(),
network_mode: "none".to_string(),
unexpected_networks: Vec::new(),
};
DockerBoundaryProvisioning {
boundary_config: BoundaryConfig {
boundary_id: self.boundary_id.clone(),
generation: self.generation.clone(),
session_id: self.session_id,
session_rotation: self.session_rotation,
auth_epoch: self.auth_epoch,
gateway_id: self.gateway_id,
verification_keys: self.verification_keys,
listener: BoundaryListener::Unix {
socket_path: self.listener_socket,
tls: self.sandbox_tls,
},
resource_claims: resource_claims.clone(),
resource_claim_files: BTreeMap::new(),
workload_identity: self.workload_identity.clone(),
driver_fence: driver_fence.clone(),
child_env: self.child_env,
},
runtime_descriptor: SandboxRuntimeDescriptor {
boundary_id: self.boundary_id,
generation: self.generation,
session_id: self.session_id,
workload_identity: self.workload_identity,
transport: SandboxTransport::Unix {
socket_path: self.control_socket,
},
tls: self.supervisor_tls,
host_gateway_ip: self.host_gateway_ip,
resource_claims,
driver_fence,
},
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn provisioning_binds_container_and_image_claims() {
let session_id = openshell_core::SandboxSessionId::new();
let tls =
openshell_sandbox_backend::boundary_protocol::generate_sandbox_tls_material(session_id)
.unwrap();
let provisioned = DockerBoundarySpec {
boundary_id: "sandbox-1".to_string(),
generation: "generation-1".to_string(),
session_id,
session_rotation: openshell_core::jwt::SessionRotation::new(1).unwrap(),
auth_epoch: openshell_core::jwt::CredentialEpoch::new(1).unwrap(),
gateway_id: "gateway-1".to_string(),
verification_keys: vec![GatewayVerificationKey {
key_id: "key-1".to_string(),
public_key_pem: "public-key".to_string(),
}],
container_id: "sha256:container".to_string(),
image_identity: "sha256:image".to_string(),
gpu_requested: true,
listener_socket: PathBuf::from("/run/openshell/boundary/control.sock"),
control_socket: PathBuf::from("/host/control.sock"),
sandbox_tls: SandboxTlsServerConfig {
certificate_chain_path: PathBuf::from("/run/openshell/boundary/server.crt"),
private_key_path: PathBuf::from("/run/openshell/boundary/server.key"),
},
supervisor_tls: SandboxTlsClientConfig {
server_name: tls.server_name,
trust_anchor_pem: tls.trust_anchor_pem,
},
host_gateway_ip: Some(IpAddr::from([127, 0, 0, 1])),
workload_identity: ResolvedWorkloadIdentity::new(
1000,
1000,
Vec::new(),
"image".to_string(),
"sha256:image".to_string(),
)
.unwrap(),
child_env: HashMap::new(),
}
.provision();
assert_eq!(
provisioned.boundary_config.resource_claims,
provisioned.runtime_descriptor.resource_claims
);
assert_eq!(
provisioned.runtime_descriptor.resource_claims["docker.container_id"],
"sha256:container"
);
assert_eq!(
provisioned.runtime_descriptor.resource_claims[GPU_RESOURCE_CLAIM],
"true"
);
assert_eq!(
provisioned.boundary_config.driver_fence,
provisioned.runtime_descriptor.driver_fence
);
assert!(
provisioned
.runtime_descriptor
.driver_fence
.validate()
.is_ok()
);
}
}
crates/openshell-driver-docker/src/lib.rs
+3074 -754
View File
File diff suppressed because it is too large Load Diff
crates/openshell-driver-docker/src/tests.rs
+693 -1131
View File
File diff suppressed because it is too large Load Diff
crates/openshell-driver-kubernetes/Cargo.toml
+4
View File
@@ -16,6 +16,8 @@ path = "src/main.rs"
[dependencies]
openshell-core = { path = "../openshell-core", default-features = false }
openshell-isolation-interface = { path = "../openshell-isolation-interface" }
openshell-sandbox-backend = { path = "../openshell-sandbox-backend" }
openshell-otel = { path = "../openshell-otel" }
openshell-policy = { path = "../openshell-policy" }
@@ -37,6 +39,8 @@ tracing-subscriber = { workspace = true }
thiserror = { workspace = true }
miette = { workspace = true }
notify = "8"
rand = { workspace = true }
rcgen = { workspace = true }
[dev-dependencies]
openshell-otel-test-support = { path = "../openshell-otel-test-support" }
crates/openshell-driver-kubernetes/README.md
+65 -64
View File
@@ -45,9 +45,52 @@ not a tenant isolation boundary.
## Runtime Model
The gateway stores platform state and delegates sandbox workload creation to
this driver. Kubernetes owns scheduling and pod lifecycle. The
`openshell-sandbox` supervisor inside each workload owns agent isolation,
credential injection, policy polling, logs, and the gateway relay.
this driver. Kubernetes owns scheduling and pod lifecycle. The workload Pod
stages the statically linked musl `openshell-sandbox` binary from
`sandbox_runtime_image`, while a directly managed Pod runs the dynamically
linked glibc `openshell-supervisor` from `supervisor_image`.
The sandbox owns the agent process, applies Landlock and child seccomp filters,
identifies the binary behind each network syscall, and relays mediated streams
to the supervisor. The supervisor authenticates to the gateway with a JWT,
loads policy and provider state, performs destination and L7 authorization, and
opens upstream connections. The workload receives no gateway credential,
provider identity socket, or corporate-proxy credential.
Both Pods run as the namespace-resolved non-root UID/GID with
`allowPrivilegeEscalation: false`, `capabilities.drop: [ALL]`, and the runtime
default seccomp profile. The sandbox installs a nested seccomp user-notification
filter without requesting a capability in the Pod spec. Startup fails closed
when the runtime blocks the required seccomp or Landlock operations.
The supervisor Pod has a direct, non-controller owner reference to the Sandbox
resource. This links its garbage-collection lifecycle to the sandbox without
competing with the Agent Sandbox controller for workload-Pod ownership.
The driver creates one namespace-wide `NetworkPolicy` before it releases any
workload Pod. It selects every OpenShell workload, denies all workload egress,
and permits OpenShell supervisor Pods to reach the sandbox TLS port. The
authenticated Sandbox Protocol binds each connection to the exact sandbox and
supervisor Pod identities. Supervisors have normal egress for gateway, DNS,
and policy-approved upstream connections unless an operator policy restricts
them. Set
`sandbox_runtime.network_policy_enforced = true` only after verifying that the cluster
CNI enforces ingress and egress `NetworkPolicy` for sandbox namespaces.
Each sandbox generation uses two immutable bootstrap Secrets. A trusted init
container stages the sandbox bootstrap into memory, and the sandbox removes it
before starting untrusted code. The other Secret is mounted only by the
supervisor. The TLS channel binds the namespace, Sandbox CR, workload Pod,
supervisor Pod, and shared network-policy identities. Stop deletes the workload
and supervisor Pods. Start rotates both Secrets and creates a new supervisor
Pod before releasing a new workload Pod. The shared network fence remains for
the lifetime of the namespace.
Kubernetes policies are additive, and the API does not attest that the CNI
enforces them. Keep sandbox namespaces administrative: untrusted principals
must not create permissive policies, create Pods, read bootstrap Secrets, or
spoof the OpenShell role labels. Exact supervisor-to-sandbox authorization is
still enforced by TLS, JWT claims, session generation, and recorded Pod UIDs.
## Sandbox Resource
@@ -95,73 +138,31 @@ mount attaches an existing PVC under `/sandbox`, which skips the default PVC.
## Credentials, TLS, and Relay
The driver injects gateway callback configuration, sandbox identity, TLS client
material, and the supervisor SSH socket path into the workload. The callback
endpoint is required because the sandbox namespace does not identify the
Gateway Service; Helm renders it from the release topology, while standalone
and raw TOML configurations must set it explicitly. Driver-owned values must
override image-provided environment variables.
Both Pods set `automountServiceAccountToken: false`. The supervisor receives an
explicit audience-bound projected token for the one-shot `IssueSandboxToken`
exchange. The driver verifies that token and the gateway returns the
sandbox-scoped JWT used by the supervisor session. The sandbox Pod receives
neither token.
Sandbox pods run as `service_account_name` and keep
`automountServiceAccountToken: false`. The only Kubernetes token exposed to the
supervisor is an explicit, audience-bound projected token mounted at
`/var/run/secrets/openshell/token` for the one-shot `IssueSandboxToken`
bootstrap exchange. The Kubernetes driver authenticates that token through the
compute-driver protocol using its own `service_account_name` and workspace-mode
namespace policy; the gateway receives only the verified sandbox ID.
The gateway uses the supervisor relay for connect, exec, logs, and file sync.
Sandbox Pods do not need direct external ingress for SSH.
The gateway uses the supervisor relay for connect, exec, and file sync. Sandbox
pods do not need direct external ingress for SSH.
The driver forwards the canonical main-process specification to the process
supervisor and sets pod `restartPolicy: Never`. Main-process environment
overrides stay local to that child; the sidecar bootstrap retains the unmodified
provider environment used by later exec, editor, and SFTP sessions.
The driver sends the canonical main-process specification only to the
supervisor. The supervisor passes admitted launch state over the protected
channel. Provider environment updates apply to future exec sessions.
## Container Security Context
The default `combined` supervisor topology grants the sandbox agent container
the Linux capabilities the supervisor needs for namespace setup and process,
filesystem, and network policy enforcement.
The sandbox, trusted bootstrap init container, and supervisor request no added
Linux capability. They run as the same numeric non-root identity, disable
privilege escalation, drop all capabilities, and inherit `RuntimeDefault`
seccomp. The sandbox and agent must use the same complete UID, GID, and
supplementary-group identity because the capability-free sandbox cannot change
credentials after launch and must inspect its same-identity descendants.
The `sidecar` supervisor topology moves pod-level network setup into a root init
container. In the default process/binary-aware mode, the long-lived network
sidecar runs as UID 0 with `allowPrivilegeEscalation: false`, drops default
Linux capabilities, and adds only `SYS_PTRACE` plus `DAC_READ_SEARCH` for
cross-UID workload `/proc` inspection. The agent container also runs as the
resolved sandbox UID/GID with `allowPrivilegeEscalation: false` and
`capabilities.drop: ["ALL"]`.
Set `sidecar.process_binary_aware_network_policy = false` to run the network
sidecar as the configured non-root `sidecar.proxy_uid`, omit the extra `/proc`
inspection capabilities, and enforce endpoint/L7 network policy without
matching `policy.binaries`.
In this mode OpenShell preserves gateway session and SSH behavior, but the
process supervisor does not perform root-to-sandbox privilege dropping or
supervisor identity mount isolation. It still applies Landlock filesystem policy
and child seccomp filters where the kernel/runtime supports them. Network
endpoint and L7 policy remain enforced by the network sidecar, and
sidecar pods use a shared process namespace so the network sidecar can resolve
process/binary identity through `/proc/<entrypoint-pid>`.
Sidecar mode keeps gateway credentials in the network sidecar. The agent
container does not mount the projected service-account token used for sandbox
token bootstrap, does not mount the sandbox client TLS secret, and does not get
gateway callback environment variables. The process supervisor receives policy
and provider environment state from the sidecar over a local control socket in
the shared sidecar state volume. The sidecar accepts only the pre-workload
process-supervisor connection, authenticates its UID/GID/PID with peer
credentials, and removes the listener afterward. SSH relays use a Linux
abstract socket whose peer PID must match that authenticated supervisor. Both
supervisors exit if the control connection closes, coupling their container
restart lifecycle before a new authoritative client can be established.
The driver uses the shared AppArmor model through `app_armor_profile`.
Supported values are `Unconfined`, `RuntimeDefault`, and
`Localhost/<profile-name>`; an empty or unset value omits
`securityContext.appArmorProfile`. Docker and Podman translate the same values
to OCI security options. Helm deployments default sandbox agent containers to
`Unconfined` because runtime/default AppArmor profiles can block the
supervisor's network namespace mount setup on AppArmor-enabled nodes.
The workload Pod does not share host network, PID, IPC, or process namespaces.
The driver uses a scheduling gate to inspect the admitted Pod and bind its UID
into the bootstrap claims before kubelet starts it.
## GPU Support
crates/openshell-driver-kubernetes/src/config.rs
+154 -436
View File
@@ -1,9 +1,8 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
pub use openshell_core::AppArmorProfile;
pub use openshell_core::DynamicStringAllowlist as OperatorNamespaceAllowlist;
use openshell_core::{ImagePullPolicy, config};
use openshell_core::config;
use serde::{Deserialize, Deserializer, Serialize};
use std::collections::BTreeMap;
#[cfg(test)]
@@ -11,6 +10,45 @@ use std::collections::BTreeSet;
use std::path::Path;
use std::str::FromStr;
/// Image pull policies accepted by the Kubernetes API.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum KubernetesImagePullPolicy {
#[serde(alias = "Always")]
Always,
#[serde(alias = "IfNotPresent")]
IfNotPresent,
#[serde(alias = "Never")]
Never,
}
impl KubernetesImagePullPolicy {
/// Return the spelling required by Kubernetes Pod specs.
#[must_use]
pub const fn as_kubernetes_str(self) -> &'static str {
match self {
Self::Always => "Always",
Self::IfNotPresent => "IfNotPresent",
Self::Never => "Never",
}
}
}
impl FromStr for KubernetesImagePullPolicy {
type Err = String;
fn from_str(value: &str) -> Result<Self, Self::Err> {
match value {
"always" | "Always" => Ok(Self::Always),
"if_not_present" | "IfNotPresent" => Ok(Self::IfNotPresent),
"never" | "Never" => Ok(Self::Never),
other => Err(format!(
"invalid Kubernetes image pull policy '{other}'; expected always, if_not_present, or never"
)),
}
}
}
/// Default gateway identity used in managed-mode namespace naming.
pub const DEFAULT_GATEWAY_ID: &str = "openshell";
@@ -23,76 +61,38 @@ pub const DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME: &str = "default";
/// Default storage size for the workspace PVC.
pub const DEFAULT_WORKSPACE_STORAGE_SIZE: &str = "2Gi";
/// Default non-root UID for relaxed Kubernetes network supervisor sidecars.
pub const DEFAULT_PROXY_UID: u32 = 1337;
/// How the supervisor binary is delivered into sandbox pods.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum SupervisorSideloadMethod {
/// Mount the supervisor OCI image directly as a read-only volume
/// (requires Kubernetes >= v1.33 with the `ImageVolume` feature gate,
/// or >= v1.36 where it is GA).
#[default]
ImageVolume,
/// Copy the binary via an init container and emptyDir volume.
/// Works on all Kubernetes versions.
InitContainer,
/// Driver-owned requirements for the Kubernetes sandbox runtime.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct KubernetesSandboxRuntimeConfig {
/// Explicit operator assertion that the cluster CNI enforces
/// `networking.k8s.io/v1` `NetworkPolicy` for the sandbox namespaces.
pub network_policy_enforced: bool,
/// TCP port exposed by the workload boundary to its paired control pod.
pub boundary_port: u16,
}
impl std::fmt::Display for SupervisorSideloadMethod {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::ImageVolume => f.write_str("image-volume"),
Self::InitContainer => f.write_str("init-container"),
impl Default for KubernetesSandboxRuntimeConfig {
fn default() -> Self {
Self {
network_policy_enforced: false,
boundary_port: 5500,
}
}
}
impl FromStr for SupervisorSideloadMethod {
type Err = String;
fn from_str(s: &str) -> Result<Self, Self::Err> {
match s {
"image-volume" => Ok(Self::ImageVolume),
"init-container" => Ok(Self::InitContainer),
other => Err(format!(
"unknown supervisor sideload method '{other}'; expected 'image-volume' or 'init-container'"
)),
impl KubernetesSandboxRuntimeConfig {
pub fn validate(&self) -> Result<(), String> {
if !self.network_policy_enforced {
return Err(
"sandbox_runtime.network_policy_enforced must be true after the operator has verified CNI NetworkPolicy enforcement"
.to_string(),
);
}
}
}
/// How the supervisor is arranged inside Kubernetes sandbox pods.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum SupervisorTopology {
/// Run networking and process supervision in the agent container.
#[default]
Combined,
/// Run network supervision in a privileged sidecar and process supervision
/// as a low-capability wrapper in the agent container.
Sidecar,
}
impl std::fmt::Display for SupervisorTopology {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::Combined => f.write_str("combined"),
Self::Sidecar => f.write_str("sidecar"),
}
}
}
impl FromStr for SupervisorTopology {
type Err = String;
fn from_str(s: &str) -> Result<Self, Self::Err> {
match s {
"combined" => Ok(Self::Combined),
"sidecar" => Ok(Self::Sidecar),
other => Err(format!("unknown topology '{other}'")),
if self.boundary_port < 1024 {
return Err("sandbox_runtime.boundary_port must be at least 1024".to_string());
}
Ok(())
}
}
@@ -138,62 +138,6 @@ impl FromStr for WorkspaceMode {
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct KubernetesSidecarConfig {
/// UID used by relaxed long-running network sidecars in `sidecar`
/// topology. The network init container installs nftables rules that
/// exempt this UID, so it must not match the sandbox workload UID.
/// Strict process/binary-aware sidecars run as UID 0 so Kubernetes grants
/// the requested `/proc` inspection capabilities into the effective set.
pub proxy_uid: u32,
/// Require process/binary-aware network policy enforcement in sidecar
/// topology. When disabled, the network sidecar runs as `proxy_uid`,
/// drops the extra `/proc` inspection permissions, and evaluates
/// endpoint/L7 policy without matching `policy.binaries`.
pub process_binary_aware_network_policy: bool,
}
impl Default for KubernetesSidecarConfig {
fn default() -> Self {
Self {
proxy_uid: DEFAULT_PROXY_UID,
process_binary_aware_network_policy: true,
}
}
}
impl KubernetesSidecarConfig {
pub fn validate_proxy_uid(&self) -> Result<(), String> {
if !(openshell_policy::MIN_SANDBOX_PROXY_UID..=openshell_policy::MAX_SANDBOX_UID)
.contains(&self.proxy_uid)
{
return Err(format!(
"sidecar.proxy_uid must be in range [{}, {}]",
openshell_policy::MIN_SANDBOX_PROXY_UID,
openshell_policy::MAX_SANDBOX_UID,
));
}
Ok(())
}
}
fn deserialize_optional_app_armor_profile<'de, D>(
deserializer: D,
) -> Result<Option<AppArmorProfile>, D::Error>
where
D: Deserializer<'de>,
{
let value = Option::<String>::deserialize(deserializer)?;
match value.as_deref() {
None | Some("") => Ok(None),
Some(value) => value
.parse::<AppArmorProfile>()
.map(Some)
.map_err(serde::de::Error::custom),
}
}
fn deserialize_provider_spiffe_workload_api_socket_path<'de, D>(
deserializer: D,
) -> Result<String, D::Error>
@@ -227,33 +171,33 @@ pub struct KubernetesComputeConfig {
/// operator mode. Hot-reloaded on change. Delivered via `ConfigMap` volume mount.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub operator_namespace_file: Option<String>,
/// Kubernetes `ServiceAccount` assigned to sandbox pods and accepted by
/// the driver's `TokenReview` bootstrap authenticator.
/// Kubernetes `ServiceAccount` assigned to the workload and supervisor Pods. Automatic
/// token mounting is disabled; only the supervisor receives an explicit
/// audience-bound projected token accepted by the bootstrap authenticator.
pub service_account_name: String,
pub default_image: String,
/// Pull policy for sandbox images. Omit to use Kubernetes's image default.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub image_pull_policy: Option<ImagePullPolicy>,
pub image_pull_policy: Option<KubernetesImagePullPolicy>,
/// Kubernetes `imagePullSecrets` names attached to sandbox pods.
pub image_pull_secrets: Vec<String>,
/// Managed-mode SSH ingress isolation. When enabled, the driver creates a
/// `NetworkPolicy` in each managed workspace namespace that permits TCP 2222
/// only from gateway pods matching this peer.
pub managed_ssh_ingress: ManagedSshIngressConfig,
/// Image that provides the `openshell-sandbox` supervisor binary.
/// Mounted directly as an image volume, or copied via an init container,
/// depending on `supervisor_sideload_method`.
pub supervisor_image: String,
/// Pull policy for the supervisor image. Omit to use Kubernetes's image
/// default.
/// Image that provides the trusted `openshell-sandbox` bootstrap binary.
pub sandbox_runtime_image: String,
/// Kubernetes `imagePullPolicy` for the sandbox runtime image.
/// When omitted, Kubernetes selects its default.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub supervisor_image_pull_policy: Option<ImagePullPolicy>,
/// How the supervisor binary is delivered into sandbox pods.
pub supervisor_sideload_method: SupervisorSideloadMethod,
/// How the supervisor is arranged for Kubernetes sandbox pods.
pub topology: SupervisorTopology,
/// Sidecar-only settings used when `topology = "sidecar"`.
pub sidecar: KubernetesSidecarConfig,
pub sandbox_runtime_image_pull_policy: Option<KubernetesImagePullPolicy>,
/// Image that provides the trusted `openshell-supervisor` control binary.
pub supervisor_image: String,
/// Kubernetes `imagePullPolicy` for the supervisor image.
/// When omitted, Kubernetes selects its default.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub supervisor_image_pull_policy: Option<KubernetesImagePullPolicy>,
/// Cross-pod sandbox/supervisor settings.
pub sandbox_runtime: KubernetesSandboxRuntimeConfig,
/// Corporate HTTP forward proxy used by the network supervisor for
/// policy-approved TLS CONNECT egress.
pub https_proxy: Option<String>,
@@ -278,14 +222,6 @@ pub struct KubernetesComputeConfig {
pub client_tls_secret_name: String,
pub host_gateway_ip: String,
pub enable_user_namespaces: bool,
/// Kubernetes `AppArmor` profile requested for the sandbox agent container.
/// Empty/None omits the `appArmorProfile` field from sandbox pod specs.
#[serde(
default,
skip_serializing_if = "Option::is_none",
deserialize_with = "deserialize_optional_app_armor_profile"
)]
pub app_armor_profile: Option<AppArmorProfile>,
pub workspace_default_storage_size: String,
/// Kubernetes `StorageClass` name for the default workspace PVC.
/// Empty string (default) = omit `storageClassName`, using the cluster's
@@ -314,13 +250,10 @@ pub struct KubernetesComputeConfig {
deserialize_with = "deserialize_provider_spiffe_workload_api_socket_path"
)]
pub provider_spiffe_workload_api_socket_path: String,
/// UID used for privilege-drop operations and workspace init container
/// ownership. The supervisor container always runs as UID 0 (root) to
/// create network namespaces and configure Landlock/seccomp; the
/// `sandbox_uid` is injected as the `SANDBOX_UID` environment variable so
/// the supervisor knows which UID to drop to for child processes.
/// Exact UID shared by `openshell-sandbox`, its agent children, the trusted
/// workspace/bootstrap init containers, and `openshell-supervisor`.
/// When empty, the driver auto-detects from `OpenShift` SCC annotations on
/// the target namespace; if those are also absent, falls back to `1000`.
/// the target namespace; if those are also absent, falls back to `10001`.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub sandbox_uid: Option<u32>,
/// GID used alongside `sandbox_uid` for PVC init container operations.
@@ -347,7 +280,7 @@ pub const MAX_SA_TOKEN_TTL_SECS: i64 = 86_400;
/// Default sandbox UID used when neither config nor `OpenShift` SCC annotations
/// provide a resolved value.
pub(crate) const DEFAULT_SANDBOX_UID: u32 = 1000;
pub(crate) const DEFAULT_SANDBOX_UID: u32 = 10001;
/// The annotation key for the `OpenShift` `ServiceAccount` UID range.
/// Format: `<start>/<size>` (e.g. `1000000000/10000`).
@@ -367,16 +300,18 @@ impl Default for KubernetesComputeConfig {
operator_namespace_file: None,
service_account_name: DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME.to_string(),
default_image: openshell_core::image::default_sandbox_image(),
// Omit the field so Kubernetes applies its own default (Always for
// `latest`, IfNotPresent otherwise).
// Default empty so the gateway omits `imagePullPolicy` from pod
// specs and Kubernetes applies its own default (Always for `latest`,
// IfNotPresent otherwise). `DEFAULT_IMAGE_PULL_POLICY` ("missing")
// is Podman vocabulary and is not a valid Kubernetes value.
image_pull_policy: None,
image_pull_secrets: Vec::new(),
managed_ssh_ingress: ManagedSshIngressConfig::default(),
sandbox_runtime_image: config::default_sandbox_runtime_image(),
sandbox_runtime_image_pull_policy: None,
supervisor_image: config::default_supervisor_image(),
supervisor_image_pull_policy: None,
supervisor_sideload_method: SupervisorSideloadMethod::default(),
topology: SupervisorTopology::default(),
sidecar: KubernetesSidecarConfig::default(),
sandbox_runtime: KubernetesSandboxRuntimeConfig::default(),
https_proxy: None,
no_proxy: None,
proxy_auth_secret_name: None,
@@ -388,7 +323,6 @@ impl Default for KubernetesComputeConfig {
client_tls_secret_name: String::new(),
host_gateway_ip: String::new(),
enable_user_namespaces: false,
app_armor_profile: None,
workspace_default_storage_size: DEFAULT_WORKSPACE_STORAGE_SIZE.to_string(),
workspace_storage_class: String::new(),
default_runtime_class_name: String::new(),
@@ -407,7 +341,6 @@ impl KubernetesComputeConfig {
self.validate_provider_spiffe_workload_api_socket_path()?;
self.validate_sandbox_identity_config()?;
self.validate_proxy_uid()?;
self.validate_image_pull_policies()?;
self.validate_upstream_proxy_config()
}
@@ -439,57 +372,20 @@ impl KubernetesComputeConfig {
}
pub fn validate_proxy_uid(&self) -> Result<(), String> {
self.sidecar.validate_proxy_uid()
}
/// Reject pull policies Kubernetes cannot express before creating pods.
pub fn validate_image_pull_policies(&self) -> Result<(), String> {
for (field, policy) in [
("image_pull_policy", self.image_pull_policy),
(
"supervisor_image_pull_policy",
self.supervisor_image_pull_policy,
),
] {
if policy == Some(ImagePullPolicy::Newer) {
return Err(format!(
"{field} = \"newer\" is supported only by the Podman compute driver"
));
}
}
Ok(())
}
/// Translate a shared policy to Kubernetes's API vocabulary.
pub fn image_pull_policy_value(policy: ImagePullPolicy) -> Result<&'static str, String> {
match policy {
ImagePullPolicy::Always => Ok("Always"),
ImagePullPolicy::IfNotPresent => Ok("IfNotPresent"),
ImagePullPolicy::Never => Ok("Never"),
ImagePullPolicy::Newer => Err(
"image pull policy 'newer' is supported only by the Podman compute driver"
.to_string(),
),
}
self.sandbox_runtime.validate()
}
/// Validate the operator-owned corporate upstream proxy configuration.
pub fn validate_upstream_proxy_config(&self) -> Result<(), String> {
use openshell_core::driver_utils::{UpstreamProxyUrlError, parse_upstream_proxy_url};
let proxy_addr = self
.https_proxy
.as_deref()
.map(|url| {
parse_upstream_proxy_url(url).map_err(|err| match err {
UpstreamProxyUrlError::Empty => {
"https_proxy must not be empty when set".to_string()
}
UpstreamProxyUrlError::InlineCredentials => "https_proxy must not embed credentials in the URL; supply them through proxy_auth_secret_name and proxy_auth_secret_key".to_string(),
err => format!("https_proxy {err}"),
})
})
.transpose()?;
if let Some(url) = &self.https_proxy {
parse_upstream_proxy_url(url).map_err(|err| match err {
UpstreamProxyUrlError::Empty => "https_proxy must not be empty when set".to_string(),
UpstreamProxyUrlError::InlineCredentials => "https_proxy must not embed credentials in the URL; supply them through proxy_auth_secret_name and proxy_auth_secret_key".to_string(),
err => format!("https_proxy {err}"),
})?;
}
if let Some(list) = self.no_proxy.as_deref() {
if list.trim().is_empty() {
@@ -548,17 +444,9 @@ impl KubernetesComputeConfig {
.to_string(),
);
}
if proxy_addr.as_ref().is_some_and(|proxy| !proxy.secure)
&& self.proxy_auth_allow_insecure != Some(true)
{
if self.proxy_auth_allow_insecure != Some(true) {
return Err("proxy credentials use cleartext Basic auth over the connection to the http:// proxy; set proxy_auth_allow_insecure = true to accept that exposure, or remove the credential Secret".to_string());
}
if self.topology == SupervisorTopology::Combined {
return Err(
"proxy credential Secrets require topology = \"sidecar\"; combined topology shares the credential mount with the workload and fsGroup can make it readable by the sandbox user"
.to_string(),
);
}
}
_ => {
return Err(
@@ -583,7 +471,7 @@ impl KubernetesComputeConfig {
/// 2. `OpenShift` SCC namespace annotations (`sa.scc.uid-range`,
/// `sa.scc.supplemental-groups`) — passed in as the optional
/// `namespace_annotations` map
/// 3. Fallback defaults: UID=`1000`, GID=UID
/// 3. Fallback defaults: UID=`10001`, GID=UID
pub fn resolve_sandbox_uid(
&self,
namespace_annotations: Option<&BTreeMap<String, String>>,
@@ -869,6 +757,51 @@ mod tests {
use super::*;
use std::collections::BTreeMap as HashMap;
#[test]
fn image_pull_policy_accepts_config_and_kubernetes_spellings() {
for (value, expected) in [
("always", KubernetesImagePullPolicy::Always),
("Always", KubernetesImagePullPolicy::Always),
("if_not_present", KubernetesImagePullPolicy::IfNotPresent),
("IfNotPresent", KubernetesImagePullPolicy::IfNotPresent),
("never", KubernetesImagePullPolicy::Never),
("Never", KubernetesImagePullPolicy::Never),
] {
assert_eq!(value.parse(), Ok(expected));
}
assert!("newer".parse::<KubernetesImagePullPolicy>().is_err());
assert!("sometimes".parse::<KubernetesImagePullPolicy>().is_err());
}
#[test]
fn image_pull_policy_fields_reject_unsupported_values() {
for field in [
"image_pull_policy",
"sandbox_runtime_image_pull_policy",
"supervisor_image_pull_policy",
] {
let input = format!("{field} = \"sometimes\"");
assert!(toml::from_str::<KubernetesComputeConfig>(&input).is_err());
}
}
#[test]
fn published_kubernetes_example_is_valid_toml() {
let docs = include_str!("../../../docs/reference/gateway-config.mdx");
let section = docs
.split_once("### Kubernetes")
.expect("Kubernetes documentation section")
.1;
let example = section
.split_once("```toml")
.expect("Kubernetes TOML fence")
.1
.split_once("```")
.expect("closed Kubernetes TOML fence")
.0;
toml::from_str::<toml::Value>(example).expect("valid Kubernetes gateway TOML example");
}
#[test]
fn default_workspace_storage_size_is_2gi() {
let cfg = KubernetesComputeConfig::default();
@@ -885,87 +818,16 @@ mod tests {
}
#[test]
fn default_topology_is_combined() {
let cfg = KubernetesComputeConfig::default();
assert_eq!(cfg.topology, SupervisorTopology::Combined);
assert_eq!(cfg.topology.to_string(), "combined");
}
#[test]
fn default_proxy_uid_is_dedicated_non_root_uid() {
let cfg = KubernetesComputeConfig::default();
assert_eq!(cfg.sidecar.proxy_uid, DEFAULT_PROXY_UID);
}
#[test]
fn default_sidecar_requires_process_binary_aware_network_policy() {
let cfg = KubernetesComputeConfig::default();
assert!(cfg.sidecar.process_binary_aware_network_policy);
}
#[test]
fn image_pull_policy_uses_shared_canonical_values() {
let cfg: KubernetesComputeConfig = serde_json::from_value(serde_json::json!({
"image_pull_policy": "if_not_present",
"supervisor_image_pull_policy": "never"
}))
.unwrap();
assert_eq!(cfg.image_pull_policy, Some(ImagePullPolicy::IfNotPresent));
assert_eq!(
cfg.supervisor_image_pull_policy,
Some(ImagePullPolicy::Never)
);
for (policy, expected) in [
(ImagePullPolicy::Always, "Always"),
(ImagePullPolicy::IfNotPresent, "IfNotPresent"),
(ImagePullPolicy::Never, "Never"),
] {
assert_eq!(
KubernetesComputeConfig::image_pull_policy_value(policy).unwrap(),
expected
);
}
}
#[test]
fn image_pull_policy_rejects_newer_for_sandbox_and_supervisor_images() {
fn sandbox_runtime_requires_network_policy_enforcement_acknowledgement() {
let mut cfg = KubernetesComputeConfig::default();
assert!(
KubernetesComputeConfig::image_pull_policy_value(ImagePullPolicy::Newer)
cfg.validate_proxy_uid()
.unwrap_err()
.contains("supported only by the Podman")
.contains("network_policy_enforced")
);
for (sandbox, supervisor) in [
(Some(ImagePullPolicy::Newer), None),
(None, Some(ImagePullPolicy::Newer)),
] {
let cfg = KubernetesComputeConfig {
image_pull_policy: sandbox,
supervisor_image_pull_policy: supervisor,
..KubernetesComputeConfig::default()
};
let error = cfg.validate_image_pull_policies().unwrap_err();
assert!(error.contains("supported only by the Podman"));
}
}
#[test]
fn serde_override_topology_sidecar() {
let json = serde_json::json!({
"topology": "sidecar"
});
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
assert_eq!(cfg.topology, SupervisorTopology::Sidecar);
}
#[test]
fn serde_override_topology_combined() {
let json = serde_json::json!({
"topology": "combined"
});
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
assert_eq!(cfg.topology, SupervisorTopology::Combined);
cfg.sandbox_runtime.network_policy_enforced = true;
cfg.validate_proxy_uid().unwrap();
}
#[test]
@@ -979,64 +841,6 @@ mod tests {
assert!(err.to_string().contains("unknown field"));
}
#[test]
fn serde_override_sidecar_process_binary_aware_network_policy_nested() {
let json = serde_json::json!({
"sidecar": {
"process_binary_aware_network_policy": false
}
});
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
assert!(!cfg.sidecar.process_binary_aware_network_policy);
}
#[test]
fn serde_override_sidecar_proxy_uid_nested() {
let json = serde_json::json!({
"sidecar": {
"proxy_uid": 2000
}
});
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
assert_eq!(cfg.sidecar.proxy_uid, 2000);
cfg.validate_proxy_uid().unwrap();
}
#[test]
fn validate_proxy_uid_rejects_privileged_uid() {
let cfg = KubernetesComputeConfig {
sidecar: KubernetesSidecarConfig {
proxy_uid: 999,
..KubernetesSidecarConfig::default()
},
..KubernetesComputeConfig::default()
};
let err = cfg.validate_proxy_uid().unwrap_err();
assert!(err.contains("proxy_uid"));
}
#[test]
fn serde_rejects_invalid_topology() {
let json = serde_json::json!({
"topology": "unsupported"
});
let err = serde_json::from_value::<KubernetesComputeConfig>(json).unwrap_err();
assert!(err.to_string().contains("unknown variant"));
}
#[test]
fn serde_rejects_removed_topology_alias_field() {
let mut json = serde_json::Map::new();
json.insert(
["supervisor", "topology"].join("_"),
serde_json::json!("sidecar"),
);
let err =
serde_json::from_value::<KubernetesComputeConfig>(serde_json::Value::Object(json))
.unwrap_err();
assert!(err.to_string().contains("unknown field"));
}
#[test]
fn serde_rejects_removed_flat_sidecar_fields() {
for json in [
@@ -1108,53 +912,6 @@ mod tests {
assert!(cfg.default_runtime_class_name.is_empty());
}
#[test]
fn default_app_armor_profile_is_none() {
let cfg = KubernetesComputeConfig::default();
assert!(cfg.app_armor_profile.is_none());
}
#[test]
fn serde_override_app_armor_profile_unconfined() {
let json = serde_json::json!({
"app_armor_profile": "Unconfined"
});
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
assert_eq!(cfg.app_armor_profile, Some(AppArmorProfile::Unconfined));
}
#[test]
fn serde_override_app_armor_profile_runtime_default() {
let json = serde_json::json!({
"app_armor_profile": "RuntimeDefault"
});
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
assert_eq!(cfg.app_armor_profile, Some(AppArmorProfile::RuntimeDefault));
}
#[test]
fn serde_override_app_armor_profile_localhost() {
let json = serde_json::json!({
"app_armor_profile": "Localhost/openshell-supervisor"
});
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
assert_eq!(
cfg.app_armor_profile,
Some(AppArmorProfile::Localhost(
"openshell-supervisor".to_string()
))
);
}
#[test]
fn serde_empty_app_armor_profile_disables_field() {
let json = serde_json::json!({
"app_armor_profile": ""
});
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
assert_eq!(cfg.app_armor_profile, None);
}
#[test]
fn serde_accepts_absolute_provider_spiffe_socket_path() {
let json = serde_json::json!({
@@ -1184,15 +941,6 @@ mod tests {
}
}
#[test]
fn serde_rejects_invalid_app_armor_profile() {
let json = serde_json::json!({
"app_armor_profile": "runtime/default"
});
let err = serde_json::from_value::<KubernetesComputeConfig>(json).unwrap_err();
assert!(err.to_string().contains("unknown AppArmor profile"));
}
#[test]
fn serde_override_image_pull_secrets() {
let json = serde_json::json!({
@@ -1399,7 +1147,6 @@ mod tests {
#[test]
fn upstream_proxy_config_accepts_secret_credentials_with_acknowledgement() {
let cfg = KubernetesComputeConfig {
topology: SupervisorTopology::Sidecar,
https_proxy: Some("http://proxy.corp.example:8080".to_string()),
proxy_auth_secret_name: Some("corporate-proxy-auth".to_string()),
proxy_auth_secret_key: Some("credentials".to_string()),
@@ -1410,22 +1157,9 @@ mod tests {
}
#[test]
fn upstream_proxy_config_accepts_tls_protected_secret_credentials() {
let cfg = KubernetesComputeConfig {
topology: SupervisorTopology::Sidecar,
https_proxy: Some("https://proxy.corp.example:8443".to_string()),
proxy_auth_secret_name: Some("corporate-proxy-auth".to_string()),
proxy_auth_secret_key: Some("credentials".to_string()),
..KubernetesComputeConfig::default()
};
assert!(cfg.validate_upstream_proxy_config().is_ok());
}
#[test]
fn toml_deserializes_sidecar_upstream_proxy_settings() {
fn toml_deserializes_upstream_proxy_settings() {
let cfg: KubernetesComputeConfig = toml::from_str(
r#"
topology = "sidecar"
https_proxy = "http://proxy.corp.example:8080"
no_proxy = ".svc.cluster.local,10.96.0.0/12"
proxy_auth_secret_name = "corporate-proxy-auth"
@@ -1547,7 +1281,6 @@ mod tests {
"bad key".to_string(), // whitespace is outside the allowed charset
] {
let cfg = KubernetesComputeConfig {
topology: SupervisorTopology::Sidecar,
https_proxy: Some("http://proxy.corp.example:8080".to_string()),
proxy_auth_secret_name: Some("corporate-proxy-auth".to_string()),
proxy_auth_secret_key: Some(key.clone()),
@@ -1565,7 +1298,6 @@ mod tests {
#[test]
fn upstream_proxy_config_accepts_max_length_secret_key() {
let cfg = KubernetesComputeConfig {
topology: SupervisorTopology::Sidecar,
https_proxy: Some("http://proxy.corp.example:8080".to_string()),
proxy_auth_secret_name: Some("corporate-proxy-auth".to_string()),
proxy_auth_secret_key: Some("a".repeat(253)),
@@ -1575,20 +1307,6 @@ mod tests {
assert!(cfg.validate_upstream_proxy_config().is_ok());
}
#[test]
fn upstream_proxy_config_rejects_credentials_in_combined_topology() {
let cfg = KubernetesComputeConfig {
topology: SupervisorTopology::Combined,
https_proxy: Some("http://proxy.corp.example:8080".to_string()),
proxy_auth_secret_name: Some("corporate-proxy-auth".to_string()),
proxy_auth_secret_key: Some("credentials".to_string()),
proxy_auth_allow_insecure: Some(true),
..KubernetesComputeConfig::default()
};
let err = cfg.validate_upstream_proxy_config().unwrap_err();
assert!(err.contains("topology = \"sidecar\""), "{err}");
}
#[test]
fn upstream_proxy_config_allows_explicit_false_acknowledgement_without_credentials() {
let cfg = KubernetesComputeConfig {
crates/openshell-driver-kubernetes/src/driver.rs
+3823 -2473
View File
File diff suppressed because it is too large Load Diff
crates/openshell-driver-kubernetes/src/grpc.rs
+37 -28
View File
@@ -158,19 +158,21 @@ impl ComputeDriver for ComputeDriverService {
&self,
request: Request<CreateSandboxRequest>,
) -> Result<Response<CreateSandboxResponse>, Status> {
self.rpc_tracer
.trace(openshell_otel::rpc::CREATE_SANDBOX, async {
let sandbox = request
.into_inner()
.sandbox
.ok_or_else(|| Status::invalid_argument("sandbox is required"))?;
self.driver
.create_sandbox(&sandbox)
.await
.map_err(|e| Status::from(openshell_core::ComputeDriverError::from(e)))?;
Ok(Response::new(CreateSandboxResponse {}))
})
.await
Box::pin(
self.rpc_tracer
.trace(openshell_otel::rpc::CREATE_SANDBOX, async {
let sandbox = request
.into_inner()
.sandbox
.ok_or_else(|| Status::invalid_argument("sandbox is required"))?;
self.driver
.create_sandbox(&sandbox)
.await
.map_err(|e| Status::from(openshell_core::ComputeDriverError::from(e)))?;
Ok(Response::new(CreateSandboxResponse {}))
}),
)
.await
}
async fn stop_sandbox(
@@ -198,21 +200,26 @@ impl ComputeDriver for ComputeDriverService {
&self,
request: Request<StartSandboxRequest>,
) -> Result<Response<StartSandboxResponse>, Status> {
self.rpc_tracer
.trace(openshell_otel::rpc::START_SANDBOX, async {
let request = request.into_inner();
if request.sandbox_id.is_empty() {
return Err(Status::invalid_argument("sandbox_id is required"));
}
self.driver
.start_sandbox(&request.sandbox_id)
Box::pin(
self.rpc_tracer
.trace(openshell_otel::rpc::START_SANDBOX, async {
let request = request.into_inner();
if request.sandbox_id.is_empty() {
return Err(Status::invalid_argument("sandbox_id is required"));
}
Box::pin(self.driver.start_sandbox(
&request.sandbox_id,
&request.generation_id,
&request.launch_authentication,
))
.await
.map_err(|error| {
Status::from(openshell_core::ComputeDriverError::from(error))
})?;
Ok(Response::new(StartSandboxResponse {}))
})
.await
Ok(Response::new(StartSandboxResponse {}))
}),
)
.await
}
async fn delete_sandbox(
@@ -250,10 +257,12 @@ impl ComputeDriver for ComputeDriverService {
let stream = stream.map(|item| item.map_err(|err| Status::internal(err.to_string())));
Ok::<ComputeDriverWatchStream, Status>(Box::pin(stream))
};
self.rpc_tracer
.trace_stream(openshell_otel::rpc::WATCH_SANDBOXES, create_stream)
.await
.map(Response::new)
Box::pin(
self.rpc_tracer
.trace_stream(openshell_otel::rpc::WATCH_SANDBOXES, create_stream),
)
.await
.map(Response::new)
}
async fn ensure_workspace(
crates/openshell-driver-kubernetes/src/isolation.rs
+447
View File
@@ -0,0 +1,447 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
//! Kubernetes provisioning for the shared authenticated boundary protocol.
//!
//! This module deliberately contains no lifecycle, process, network, identity,
//! or wire implementation. The driver places the sandbox runtime, binds
//! immutable Kubernetes resource identities, and provisions TCP coordinates;
//! `openshell-isolation-interface` and `openshell-sandbox` provide the common
//! control and boundary behavior.
use std::collections::{BTreeMap, HashMap};
use std::net::{IpAddr, SocketAddr};
use std::path::PathBuf;
use k8s_openapi::api::networking::v1::{
NetworkPolicy, NetworkPolicyEgressRule, NetworkPolicyIngressRule, NetworkPolicyPeer,
NetworkPolicyPort, NetworkPolicySpec,
};
use k8s_openapi::apimachinery::pkg::apis::meta::v1::LabelSelector;
use k8s_openapi::apimachinery::pkg::util::intstr::IntOrString;
use kube::core::ObjectMeta;
use openshell_isolation_interface::contract::{DriverFenceEvidence, ResolvedWorkloadIdentity};
use openshell_sandbox_backend::boundary_protocol::{
BoundaryConfig, BoundaryListener, GatewayVerificationKey, SandboxRuntimeDescriptor,
SandboxTlsClientConfig, SandboxTlsServerConfig, SandboxTransport,
};
/// Isolation backend implemented by the `OpenShell` sandbox runtime.
pub const BACKEND_NAME: &str = openshell_sandbox_backend::BACKEND_NAME;
/// Label that binds the workload and supervisor pods in one unique pair.
pub const BOUNDARY_PAIR_LABEL: &str = "openshell.ai/boundary-pair";
/// Label distinguishing the two pods in a sandbox generation.
pub const BOUNDARY_ROLE_LABEL: &str = "openshell.ai/boundary-role";
const WORKLOAD_ROLE: &str = "workload";
const SUPERVISOR_ROLE: &str = "supervisor";
/// Driver-owned inputs for the workload pod's Kubernetes network fence.
///
/// This is the first phase of sandbox-runtime provisioning. The driver applies the
/// returned labels to the respective pods and creates the returned policy. It
/// then observes the policy UID and resourceVersion and supplies both to
/// `KubernetesSandboxRuntimeBoundarySpec`.
pub struct KubernetesSandboxRuntimeNetworkFenceSpec {
pub namespace: String,
pub policy_name: String,
pub supervisor_policy_name: String,
pub boundary_port: u16,
}
/// Labels and policy needed to remove direct workload-pod egress.
pub struct KubernetesSandboxRuntimeNetworkFence {
pub workload_labels: BTreeMap<String, String>,
pub control_labels: BTreeMap<String, String>,
pub workload_policy: NetworkPolicy,
pub supervisor_policy: NetworkPolicy,
}
impl KubernetesSandboxRuntimeNetworkFenceSpec {
/// Render the namespace-wide workload fence.
///
/// Kubernetes `NetworkPolicy` is connection-aware: traffic returning over
/// the control-initiated boundary connection is allowed even though the
/// workload pod has no egress rules. The control pod remains responsible
/// for opening policy-approved upstream connections.
#[must_use]
pub fn provision(self) -> KubernetesSandboxRuntimeNetworkFence {
let workload_labels = role_labels(WORKLOAD_ROLE);
let control_labels = role_labels(SUPERVISOR_ROLE);
let workload_policy = NetworkPolicy {
metadata: ObjectMeta {
name: Some(self.policy_name),
namespace: Some(self.namespace.clone()),
..Default::default()
},
spec: Some(NetworkPolicySpec {
pod_selector: LabelSelector {
match_labels: Some(workload_labels.clone()),
..Default::default()
},
policy_types: Some(vec!["Ingress".to_string(), "Egress".to_string()]),
// Any trusted OpenShell supervisor in this namespace may
// reach a sandbox listener. The Sandbox Protocol enforces the
// exact sandbox, generation, and Pod UID binding.
ingress: Some(vec![NetworkPolicyIngressRule {
from: Some(vec![NetworkPolicyPeer {
pod_selector: Some(LabelSelector {
match_labels: Some(control_labels.clone()),
..Default::default()
}),
..Default::default()
}]),
ports: Some(vec![NetworkPolicyPort {
port: Some(IntOrString::Int(i32::from(self.boundary_port))),
protocol: Some("TCP".to_string()),
..Default::default()
}]),
}]),
// An explicit empty list selects the pod for egress and allows
// no new workload-initiated connections, including DNS and the
// Kubernetes API. Reply traffic for allowed ingress remains
// permitted by conforming NetworkPolicy implementations.
egress: Some(Vec::new()),
}),
};
// Namespace-wide default-deny policies are additive with this rule.
// Select only OpenShell supervisor pods and explicitly allow their
// policy-approved DNS and upstream connections.
let supervisor_policy = NetworkPolicy {
metadata: ObjectMeta {
name: Some(self.supervisor_policy_name),
namespace: Some(self.namespace),
..Default::default()
},
spec: Some(NetworkPolicySpec {
pod_selector: LabelSelector {
match_labels: Some(control_labels.clone()),
..Default::default()
},
policy_types: Some(vec!["Egress".to_string()]),
egress: Some(vec![NetworkPolicyEgressRule::default()]),
..Default::default()
}),
};
KubernetesSandboxRuntimeNetworkFence {
workload_labels,
control_labels,
workload_policy,
supervisor_policy,
}
}
}
fn role_labels(role: &str) -> BTreeMap<String, String> {
BTreeMap::from([(BOUNDARY_ROLE_LABEL.to_string(), role.to_string())])
}
/// Driver-owned inputs that bind one workload/supervisor pair to one boundary.
///
/// The driver constructs this only after Kubernetes has assigned every UID and
/// after it has observed the namespace workload-policy resource version. The
/// workload stays held until the matching boundary config and supervisor
/// resources have been installed.
pub struct KubernetesSandboxRuntimeBoundarySpec {
pub boundary_id: String,
pub generation: String,
pub session_id: openshell_core::SandboxSessionId,
pub session_rotation: openshell_core::jwt::SessionRotation,
pub auth_epoch: openshell_core::jwt::CredentialEpoch,
pub gateway_id: String,
pub verification_keys: Vec<GatewayVerificationKey>,
pub namespace_uid: String,
pub sandbox_resource_uid: String,
pub workload_pod_uid: String,
pub workload_pod_uid_path: PathBuf,
pub supervisor_pod_uid: String,
pub egress_policy_uid: String,
pub egress_policy_resource_version: String,
pub boundary_listener: SocketAddr,
pub control_authority: String,
pub control_address: SocketAddr,
pub sandbox_tls: SandboxTlsServerConfig,
pub supervisor_tls: SandboxTlsClientConfig,
pub host_gateway_ip: Option<IpAddr>,
pub workload_identity: ResolvedWorkloadIdentity,
pub child_env: HashMap<String, String>,
}
/// Protected workload-pod config and matching sandbox-runtime descriptor.
pub struct KubernetesSandboxRuntimeBoundaryProvisioning {
pub boundary_config: BoundaryConfig,
pub runtime_descriptor: SandboxRuntimeDescriptor,
}
impl KubernetesSandboxRuntimeBoundarySpec {
/// Produce both sides of the common protocol from one observed Kubernetes
/// resource set so a stale or recreated object cannot be attached.
#[must_use]
pub fn provision(self) -> KubernetesSandboxRuntimeBoundaryProvisioning {
let resource_claims = BTreeMap::from([
("kubernetes.namespace_uid".to_string(), self.namespace_uid),
(
"kubernetes.sandbox_resource_uid".to_string(),
self.sandbox_resource_uid,
),
(
"kubernetes.workload_pod_uid".to_string(),
self.workload_pod_uid,
),
(
"kubernetes.supervisor_pod_uid".to_string(),
self.supervisor_pod_uid,
),
(
"kubernetes.egress_policy_uid".to_string(),
self.egress_policy_uid,
),
(
"kubernetes.egress_policy_resource_version".to_string(),
self.egress_policy_resource_version,
),
]);
let driver_fence = DriverFenceEvidence::Kubernetes {
network_policy_uid: resource_claims["kubernetes.egress_policy_uid"].clone(),
network_policy_resource_version:
resource_claims["kubernetes.egress_policy_resource_version"].clone(),
ingress_isolated: true,
egress_isolated: true,
egress_rule_count: 0,
};
KubernetesSandboxRuntimeBoundaryProvisioning {
boundary_config: BoundaryConfig {
boundary_id: self.boundary_id.clone(),
generation: self.generation.clone(),
session_id: self.session_id,
session_rotation: self.session_rotation,
auth_epoch: self.auth_epoch,
gateway_id: self.gateway_id,
verification_keys: self.verification_keys,
listener: BoundaryListener::TlsTcp {
address: self.boundary_listener,
tls: self.sandbox_tls,
},
resource_claims: resource_claims.clone(),
resource_claim_files: BTreeMap::from([(
"kubernetes.workload_pod_uid".to_string(),
self.workload_pod_uid_path,
)]),
workload_identity: self.workload_identity.clone(),
driver_fence: driver_fence.clone(),
child_env: self.child_env,
},
runtime_descriptor: SandboxRuntimeDescriptor {
boundary_id: self.boundary_id,
generation: self.generation,
session_id: self.session_id,
workload_identity: self.workload_identity,
transport: SandboxTransport::Tcp {
authority: self.control_authority,
addresses: vec![self.control_address],
},
tls: self.supervisor_tls,
host_gateway_ip: self.host_gateway_ip,
resource_claims,
driver_fence,
},
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn spec() -> KubernetesSandboxRuntimeBoundarySpec {
KubernetesSandboxRuntimeBoundarySpec {
boundary_id: "sandbox-1".to_string(),
generation: "generation-1".to_string(),
session_id: openshell_core::SandboxSessionId::new(),
session_rotation: openshell_core::jwt::SessionRotation::new(1).unwrap(),
auth_epoch: openshell_core::jwt::CredentialEpoch::new(1).unwrap(),
gateway_id: "gateway-1".to_string(),
verification_keys: vec![GatewayVerificationKey {
key_id: "key-1".to_string(),
public_key_pem: "public-key".to_string(),
}],
namespace_uid: "namespace-uid".to_string(),
sandbox_resource_uid: "sandbox-resource-uid".to_string(),
workload_pod_uid: "pod-uid".to_string(),
workload_pod_uid_path: PathBuf::from("/.openshell/pod-identity/uid"),
supervisor_pod_uid: "supervisor-pod-uid".to_string(),
egress_policy_uid: "network-policy-uid".to_string(),
egress_policy_resource_version: "1945".to_string(),
boundary_listener: "0.0.0.0:5500".parse().expect("valid listener"),
control_authority: "os-boundary-sandbox.default.svc:5500".to_string(),
control_address: "10.42.0.7:5500".parse().expect("valid target"),
sandbox_tls: SandboxTlsServerConfig {
certificate_chain_path: PathBuf::from("/run/boundary/tls.crt"),
private_key_path: PathBuf::from("/run/boundary/tls.key"),
},
supervisor_tls: SandboxTlsClientConfig {
server_name: "boundary.sandbox.openshell".to_string(),
trust_anchor_pem: "test-ca".to_string(),
},
host_gateway_ip: Some("10.42.0.1".parse().expect("valid gateway IP")),
workload_identity: ResolvedWorkloadIdentity::new(
1000,
1000,
vec![1000],
"kubernetes-config".to_string(),
"sandbox:sandbox-resource-uid".to_string(),
)
.unwrap(),
child_env: HashMap::new(),
}
}
#[test]
fn provisioning_binds_identical_kubernetes_resource_claims() {
let provisioned = spec().provision();
assert_eq!(
provisioned.boundary_config.resource_claims,
provisioned.runtime_descriptor.resource_claims
);
assert_eq!(
provisioned.runtime_descriptor.resource_claims["kubernetes.sandbox_resource_uid"],
"sandbox-resource-uid"
);
assert_eq!(
provisioned.runtime_descriptor.resource_claims["kubernetes.egress_policy_resource_version"],
"1945"
);
assert_eq!(
provisioned.boundary_config.driver_fence,
provisioned.runtime_descriptor.driver_fence
);
assert!(
provisioned
.runtime_descriptor
.driver_fence
.validate()
.is_ok()
);
}
#[test]
fn provisioning_uses_one_shared_tcp_protocol_across_pods() {
let provisioned = spec().provision();
assert_eq!(
provisioned.boundary_config.listener,
BoundaryListener::TlsTcp {
address: "0.0.0.0:5500".parse().expect("valid listener"),
tls: SandboxTlsServerConfig {
certificate_chain_path: PathBuf::from("/run/boundary/tls.crt"),
private_key_path: PathBuf::from("/run/boundary/tls.key"),
},
}
);
assert_eq!(
provisioned.runtime_descriptor.transport,
SandboxTransport::Tcp {
authority: "os-boundary-sandbox.default.svc:5500".to_string(),
addresses: vec!["10.42.0.7:5500".parse().expect("valid target")],
}
);
assert_eq!(
provisioned.runtime_descriptor.tls,
SandboxTlsClientConfig {
server_name: "boundary.sandbox.openshell".to_string(),
trust_anchor_pem: "test-ca".to_string(),
}
);
}
#[test]
fn network_fence_denies_all_workload_initiated_egress() {
let fence = KubernetesSandboxRuntimeNetworkFenceSpec {
namespace: "sandbox-ns".to_string(),
policy_name: "openshell-boundary-sandbox-1".to_string(),
supervisor_policy_name: "openshell-sandbox-supervisors".to_string(),
boundary_port: 5500,
}
.provision();
let policy_spec = fence.workload_policy.spec.expect("policy has a spec");
assert_eq!(
policy_spec.policy_types,
Some(vec!["Ingress".to_string(), "Egress".to_string()])
);
assert_eq!(policy_spec.egress, Some(Vec::new()));
assert_eq!(
policy_spec.pod_selector.match_labels,
Some(fence.workload_labels)
);
}
#[test]
fn network_fence_allows_namespace_supervisors_to_boundary_port() {
let fence = KubernetesSandboxRuntimeNetworkFenceSpec {
namespace: "sandbox-ns".to_string(),
policy_name: "openshell-boundary-sandbox-1".to_string(),
supervisor_policy_name: "openshell-sandbox-supervisors".to_string(),
boundary_port: 5500,
}
.provision();
let policy_spec = fence.workload_policy.spec.expect("policy has a spec");
let ingress = policy_spec
.ingress
.expect("policy has ingress rules")
.pop()
.expect("policy has one ingress rule");
let peer = ingress
.from
.expect("rule has peers")
.pop()
.expect("rule has one peer");
assert_eq!(
peer.pod_selector
.expect("peer has a pod selector")
.match_labels,
Some(fence.control_labels)
);
assert!(peer.namespace_selector.is_none());
let port = ingress
.ports
.expect("rule has ports")
.pop()
.expect("rule has one port");
assert_eq!(port.protocol.as_deref(), Some("TCP"));
assert_eq!(port.port, Some(IntOrString::Int(5500)));
}
#[test]
fn network_fence_keeps_supervisor_egress_available() {
let fence = KubernetesSandboxRuntimeNetworkFenceSpec {
namespace: "sandbox-ns".to_string(),
policy_name: "openshell-sandbox-workloads".to_string(),
supervisor_policy_name: "openshell-sandbox-supervisors".to_string(),
boundary_port: 5500,
}
.provision();
let policy_spec = fence
.supervisor_policy
.spec
.expect("supervisor policy has a spec");
assert_eq!(policy_spec.policy_types, Some(vec!["Egress".to_string()]));
assert_eq!(
policy_spec.pod_selector.match_labels,
Some(fence.control_labels)
);
assert_eq!(
policy_spec.egress,
Some(vec![NetworkPolicyEgressRule::default()])
);
}
}
crates/openshell-driver-kubernetes/src/lib.rs
+5 -4
View File
@@ -4,13 +4,14 @@
pub mod config;
pub mod driver;
pub mod grpc;
pub mod isolation;
pub mod otel_tracing;
mod sandbox_runtime;
pub use config::{
AppArmorProfile, DEFAULT_GATEWAY_ID, DEFAULT_PROXY_UID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME,
DEFAULT_WORKSPACE_STORAGE_SIZE, KubernetesComputeConfig, KubernetesSidecarConfig,
ManagedSshIngressConfig, SupervisorSideloadMethod, SupervisorTopology, WorkspaceMode,
managed_namespace_prefix,
DEFAULT_GATEWAY_ID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, DEFAULT_WORKSPACE_STORAGE_SIZE,
KubernetesComputeConfig, KubernetesImagePullPolicy, KubernetesSandboxRuntimeConfig,
ManagedSshIngressConfig, WorkspaceMode, managed_namespace_prefix,
};
pub use driver::{KubernetesComputeDriver, KubernetesDriverError};
pub use grpc::ComputeDriverService;
crates/openshell-driver-kubernetes/src/main.rs
+28 -49
View File
@@ -8,13 +8,12 @@ use std::net::SocketAddr;
use std::path::PathBuf;
use tracing::info;
use openshell_core::VERSION;
use openshell_core::proto::compute::v1::compute_driver_server::ComputeDriverServer;
use openshell_core::{ImagePullPolicy, VERSION};
use openshell_driver_kubernetes::{
AppArmorProfile, ComputeDriverService, DEFAULT_GATEWAY_ID, DEFAULT_PROXY_UID,
DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, KubernetesComputeConfig, KubernetesComputeDriver,
KubernetesSidecarConfig, ManagedSshIngressConfig, SupervisorSideloadMethod, SupervisorTopology,
WorkspaceMode,
ComputeDriverService, DEFAULT_GATEWAY_ID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME,
KubernetesComputeConfig, KubernetesComputeDriver, KubernetesImagePullPolicy,
KubernetesSandboxRuntimeConfig, ManagedSshIngressConfig, WorkspaceMode,
};
#[derive(Parser, Debug)]
@@ -72,7 +71,7 @@ struct Args {
sandbox_image: Option<String>,
#[arg(long, env = "OPENSHELL_SANDBOX_IMAGE_PULL_POLICY")]
sandbox_image_pull_policy: Option<ImagePullPolicy>,
sandbox_image_pull_policy: Option<KubernetesImagePullPolicy>,
#[arg(
long,
@@ -94,10 +93,8 @@ struct Args {
)]
managed_ssh_gateway_pod_selector: Vec<String>,
/// Gateway callback endpoint reachable from sandbox pods. Kubernetes
/// service topology cannot be inferred from the sandbox namespace.
#[arg(long, env = "OPENSHELL_GRPC_ENDPOINT")]
grpc_endpoint: String,
grpc_endpoint: Option<String>,
#[arg(
long,
@@ -112,37 +109,31 @@ struct Args {
#[arg(long, env = "OPENSHELL_HOST_GATEWAY_IP")]
host_gateway_ip: Option<String>,
#[arg(long, env = "OPENSHELL_SANDBOX_RUNTIME_IMAGE")]
sandbox_runtime_image: Option<String>,
#[arg(long, env = "OPENSHELL_SANDBOX_RUNTIME_IMAGE_PULL_POLICY")]
sandbox_runtime_image_pull_policy: Option<KubernetesImagePullPolicy>,
#[arg(long, env = "OPENSHELL_SUPERVISOR_IMAGE")]
supervisor_image: Option<String>,
#[arg(long, env = "OPENSHELL_SUPERVISOR_IMAGE_PULL_POLICY")]
supervisor_image_pull_policy: Option<ImagePullPolicy>,
supervisor_image_pull_policy: Option<KubernetesImagePullPolicy>,
#[arg(
long,
env = "OPENSHELL_SUPERVISOR_SIDELOAD_METHOD",
default_value = "image-volume"
env = "OPENSHELL_K8S_SANDBOX_RUNTIME_NETWORK_POLICY_ENFORCED",
default_value_t = false
)]
supervisor_sideload_method: SupervisorSideloadMethod,
#[arg(long, env = "OPENSHELL_K8S_TOPOLOGY", default_value = "combined")]
topology: SupervisorTopology,
sandbox_runtime_network_policy_enforced: bool,
#[arg(
long = "sidecar-proxy-uid",
alias = "proxy-uid",
env = "OPENSHELL_K8S_SIDECAR_PROXY_UID",
default_value_t = DEFAULT_PROXY_UID
long,
env = "OPENSHELL_K8S_SANDBOX_RUNTIME_BOUNDARY_PORT",
default_value_t = 5500
)]
sidecar_proxy_uid: u32,
#[arg(
long = "sidecar-process-binary-aware-network-policy",
env = "OPENSHELL_K8S_SIDECAR_PROCESS_BINARY_AWARE_NETWORK_POLICY",
default_value_t = true,
action = ArgAction::Set
)]
sidecar_process_binary_aware_network_policy: bool,
sandbox_runtime_boundary_port: u16,
/// Corporate HTTP forward proxy for policy-approved TLS CONNECT egress.
#[arg(long, env = "OPENSHELL_UPSTREAM_PROXY")]
@@ -171,9 +162,6 @@ struct Args {
#[arg(long, env = "OPENSHELL_ENABLE_USER_NAMESPACES")]
enable_user_namespaces: bool,
#[arg(long, env = "OPENSHELL_K8S_APP_ARMOR_PROFILE")]
app_armor_profile: Option<AppArmorProfile>,
/// Lifetime (seconds) of the projected `ServiceAccount` token
/// kubelet writes into each sandbox pod for the `IssueSandboxToken`
/// bootstrap exchange. Kubelet enforces a minimum of 600s; the
@@ -257,16 +245,17 @@ async fn main() -> Result<()> {
gateway_namespace: args.managed_ssh_gateway_namespace.unwrap_or_default(),
gateway_pod_selector: managed_ssh_gateway_pod_selector,
},
sandbox_runtime_image: args
.sandbox_runtime_image
.unwrap_or_else(openshell_core::config::default_sandbox_runtime_image),
sandbox_runtime_image_pull_policy: args.sandbox_runtime_image_pull_policy,
supervisor_image: args
.supervisor_image
.unwrap_or_else(openshell_core::config::default_supervisor_image),
supervisor_image_pull_policy: args.supervisor_image_pull_policy,
supervisor_sideload_method: args.supervisor_sideload_method,
topology: args.topology,
sidecar: KubernetesSidecarConfig {
proxy_uid: args.sidecar_proxy_uid,
process_binary_aware_network_policy: args
.sidecar_process_binary_aware_network_policy,
sandbox_runtime: KubernetesSandboxRuntimeConfig {
network_policy_enforced: args.sandbox_runtime_network_policy_enforced,
boundary_port: args.sandbox_runtime_boundary_port,
},
https_proxy: args.https_proxy,
no_proxy: args.no_proxy,
@@ -274,12 +263,11 @@ async fn main() -> Result<()> {
proxy_auth_secret_key: args.proxy_auth_secret_key,
proxy_auth_allow_insecure: args.proxy_auth_allow_insecure.then_some(true),
proxy_connect_by_hostname: args.proxy_connect_by_hostname.then_some(true),
grpc_endpoint: args.grpc_endpoint,
grpc_endpoint: args.grpc_endpoint.unwrap_or_default(),
ssh_socket_path: args.sandbox_ssh_socket_path,
client_tls_secret_name: args.client_tls_secret_name.unwrap_or_default(),
host_gateway_ip: args.host_gateway_ip.unwrap_or_default(),
enable_user_namespaces: args.enable_user_namespaces,
app_armor_profile: args.app_armor_profile,
workspace_default_storage_size: std::env::var(
"OPENSHELL_K8S_WORKSPACE_DEFAULT_STORAGE_SIZE",
)
@@ -337,13 +325,6 @@ async fn main() -> Result<()> {
mod tests {
use super::*;
#[test]
fn requires_explicit_gateway_callback_endpoint() {
let error = Args::try_parse_from(["openshell-driver-kubernetes"])
.expect_err("Kubernetes service topology must be explicit");
assert!(error.to_string().contains("--grpc-endpoint"));
}
#[test]
fn accepts_gateway_otlp_configuration() {
let args = Args::try_parse_from([
@@ -352,8 +333,6 @@ mod tests {
"http://collector.example:4317",
"--gateway-name",
"kubernetes-dev",
"--grpc-endpoint",
"http://openshell.example:8080",
])
.expect("OTLP endpoint should parse");
crates/openshell-driver-kubernetes/src/sandbox_runtime.rs
+822
View File
@@ -0,0 +1,822 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
//! Kubernetes resources that place and protect the `OpenShell` sandbox runtime.
use std::collections::BTreeMap;
use std::path::Path;
use k8s_openapi::ByteString;
use k8s_openapi::api::core::v1::{
CSIVolumeSource, Capabilities, Container, EmptyDirVolumeSource, EnvVar, ExecAction, KeyToPath,
LocalObjectReference, Pod, PodSchedulingGate, PodSecurityContext, PodSpec, Probe,
ProjectedVolumeSource, Secret, SecretVolumeSource, SecurityContext, Service,
ServiceAccountTokenProjection, ServicePort, ServiceSpec, Volume, VolumeMount, VolumeProjection,
};
use k8s_openapi::apimachinery::pkg::apis::meta::v1::OwnerReference;
use k8s_openapi::apimachinery::pkg::util::intstr::IntOrString;
use kube::core::ObjectMeta;
use rcgen::{CertificateParams, DnType, IsCa, KeyPair, KeyUsagePurpose};
use crate::isolation::{
BOUNDARY_PAIR_LABEL, BOUNDARY_ROLE_LABEL, KubernetesSandboxRuntimeNetworkFence,
KubernetesSandboxRuntimeNetworkFenceSpec,
};
pub const SANDBOX_SECRET_COMPONENT: &str = "sandbox-bootstrap";
pub const SUPERVISOR_SECRET_COMPONENT: &str = "supervisor-bootstrap";
pub const BOUNDARY_CONFIG_KEY: &str = "boundary.json";
pub const BACKEND_DESCRIPTOR_KEY: &str = "runtime-descriptor.json";
pub const BOUNDARY_CERTIFICATE_KEY: &str = "tls.crt";
pub const BOUNDARY_PRIVATE_KEY: &str = "tls.key";
pub const SUPERVISOR_AUTH_BUNDLE_KEY: &str = "auth.json";
pub const PROXY_CA_CERTIFICATE_KEY: &str = "proxy-ca.crt";
pub const PROXY_CA_PRIVATE_KEY: &str = "proxy-ca.key";
pub const SANDBOX_BOOTSTRAP_INPUT_PATH: &str = "/.openshell/bootstrap-input";
pub const BOUNDARY_CONFIG_PATH: &str = "/.openshell/state/bootstrap/boundary.json";
pub const BOUNDARY_CERTIFICATE_PATH: &str = "/.openshell/state/bootstrap/tls.crt";
pub const BOUNDARY_PRIVATE_KEY_PATH: &str = "/.openshell/state/bootstrap/tls.key";
pub const BACKEND_DESCRIPTOR_PATH: &str = "/.openshell/supervisor/runtime-descriptor.json";
pub const SUPERVISOR_AUTH_BUNDLE_PATH: &str = "/.openshell/supervisor/auth.json";
pub const PROXY_CA_CERTIFICATE_PATH: &str = "/.openshell/supervisor/proxy-ca.crt";
pub const PROXY_CA_PRIVATE_KEY_PATH: &str = "/.openshell/supervisor/proxy-ca.key";
pub const CONTROL_HEALTH_SOCKET_PATH: &str = "/run/openshell/health.sock";
pub const NAMESPACE_WORKLOAD_POLICY_NAME: &str = "openshell-sandbox-workloads";
pub const NAMESPACE_SUPERVISOR_EGRESS_POLICY_NAME: &str = "openshell-sandbox-supervisors";
pub struct ProxyCaMaterial {
pub certificate_pem: String,
pub private_key_pem: String,
}
pub fn generate_proxy_ca_material() -> Result<ProxyCaMaterial, String> {
let key = KeyPair::generate().map_err(|error| format!("generate proxy CA key: {error}"))?;
let mut params = CertificateParams::default();
params.is_ca = IsCa::Ca(rcgen::BasicConstraints::Unconstrained);
params
.distinguished_name
.push(DnType::CommonName, "OpenShell Sandbox CA");
params
.distinguished_name
.push(DnType::OrganizationName, "OpenShell");
params.key_usages = vec![KeyUsagePurpose::KeyCertSign, KeyUsagePurpose::CrlSign];
let certificate = params
.self_signed(&key)
.map_err(|error| format!("generate proxy CA certificate: {error}"))?;
Ok(ProxyCaMaterial {
certificate_pem: certificate.pem(),
private_key_pem: key.serialize_pem(),
})
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct SandboxRuntimeNames {
pub sandbox_secret: String,
pub supervisor_secret: String,
pub boundary_service: String,
pub supervisor_pod: String,
pub workload_policy: String,
pub supervisor_policy: String,
}
impl SandboxRuntimeNames {
#[must_use]
pub fn new(sandbox_id: &str) -> Self {
let suffix = sandbox_id.to_ascii_lowercase();
Self {
sandbox_secret: format!("os-sandbox-{suffix}"),
supervisor_secret: format!("os-supervisor-{suffix}"),
boundary_service: format!("os-boundary-{suffix}"),
supervisor_pod: format!("os-supervisor-{suffix}"),
workload_policy: NAMESPACE_WORKLOAD_POLICY_NAME.to_string(),
supervisor_policy: NAMESPACE_SUPERVISOR_EGRESS_POLICY_NAME.to_string(),
}
}
/// Return stable companion names plus generation-specific immutable
/// bootstrap Secret names.
#[must_use]
pub fn for_generation(sandbox_id: &str, generation: &str) -> Self {
let mut names = Self::new(sandbox_id);
let generation = generation
.chars()
.filter(char::is_ascii_alphanumeric)
.take(12)
.collect::<String>()
.to_ascii_lowercase();
names.sandbox_secret = format!("{}-{generation}", names.sandbox_secret);
names.supervisor_secret = format!("{}-{generation}", names.supervisor_secret);
names
}
}
#[must_use]
pub fn pair_label_value(sandbox_id: &str) -> String {
sandbox_id.to_ascii_lowercase()
}
#[must_use]
pub fn workload_fence(
namespace: &str,
names: &SandboxRuntimeNames,
boundary_port: u16,
) -> KubernetesSandboxRuntimeNetworkFence {
KubernetesSandboxRuntimeNetworkFenceSpec {
namespace: namespace.to_string(),
policy_name: names.workload_policy.clone(),
supervisor_policy_name: names.supervisor_policy.clone(),
boundary_port,
}
.provision()
}
#[must_use]
pub fn boundary_service(
namespace: &str,
names: &SandboxRuntimeNames,
sandbox_id: &str,
boundary_port: u16,
owner: OwnerReference,
) -> Service {
Service {
metadata: ObjectMeta {
name: Some(names.boundary_service.clone()),
namespace: Some(namespace.to_string()),
owner_references: Some(vec![owner]),
labels: Some(common_labels(sandbox_id, "boundary-service")),
..Default::default()
},
spec: Some(ServiceSpec {
selector: Some(pair_labels(sandbox_id, "workload")),
ports: Some(vec![ServicePort {
name: Some("boundary".to_string()),
protocol: Some("TCP".to_string()),
port: i32::from(boundary_port),
target_port: Some(IntOrString::Int(i32::from(boundary_port))),
..Default::default()
}]),
..Default::default()
}),
..Default::default()
}
}
#[allow(clippy::too_many_arguments, clippy::similar_names)]
pub fn supervisor_pod(
namespace: &str,
names: &SandboxRuntimeNames,
sandbox_id: &str,
sandbox_name: &str,
gateway_id: &str,
supervisor_image: &str,
supervisor_pull_policy: Option<crate::KubernetesImagePullPolicy>,
service_account_name: &str,
control_uid: u32,
control_gid: u32,
image_pull_secrets: &[String],
grpc_endpoint: &str,
client_tls_secret_name: &str,
main_process_spec: &str,
log_level: &str,
sa_token_ttl_secs: i64,
https_proxy: Option<&str>,
no_proxy: Option<&str>,
proxy_auth_secret: Option<(&str, &str)>,
proxy_auth_allow_insecure: bool,
proxy_connect_by_hostname: bool,
provider_spiffe_socket_path: Option<&str>,
owner: OwnerReference,
) -> Result<Pod, String> {
let labels = control_labels(sandbox_id, gateway_id);
let mut environment = vec![
env_var(
"OPENSHELL_ADMITTED_ISOLATION_BACKEND",
crate::isolation::BACKEND_NAME,
),
env_var("OPENSHELL_ENDPOINT", grpc_endpoint),
env_var("OPENSHELL_SANDBOX_ID", sandbox_id),
env_var("OPENSHELL_SANDBOX", sandbox_name),
env_var("OPENSHELL_MAIN_PROCESS_SPEC", main_process_spec),
env_var(
"OPENSHELL_K8S_SA_TOKEN_FILE",
"/var/run/secrets/openshell/token",
),
env_var("OPENSHELL_SSH_SOCKET_PATH", "/run/openshell/ssh.sock"),
env_var(openshell_core::sandbox_env::SSH_SOCKET_SHARED, "true"),
env_var("OPENSHELL_PROXY_TLS_DIR", "/run/openshell/proxy-tls"),
env_var(
openshell_core::sandbox_env::PROXY_CA_CERT,
PROXY_CA_CERTIFICATE_PATH,
),
env_var(
openshell_core::sandbox_env::PROXY_CA_KEY,
PROXY_CA_PRIVATE_KEY_PATH,
),
env_var("OPENSHELL_LOG_LEVEL", log_level),
env_var(
openshell_core::sandbox_env::TELEMETRY_ENABLED,
openshell_core::telemetry::enabled_env_value(),
),
env_var(
openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES,
"",
),
];
let mut volume_mounts = vec![
volume_mount("bootstrap", "/.openshell/supervisor", true),
volume_mount("sa-token", "/var/run/secrets/openshell", true),
volume_mount("run", "/run/openshell", false),
volume_mount("logs", "/var/log", false),
];
let mut volumes = vec![
secret_volume("bootstrap", &names.supervisor_secret, None),
Volume {
name: "sa-token".to_string(),
projected: Some(ProjectedVolumeSource {
default_mode: Some(0o440),
sources: Some(vec![VolumeProjection {
service_account_token: Some(ServiceAccountTokenProjection {
audience: Some("openshell-gateway".to_string()),
expiration_seconds: Some(sa_token_ttl_secs),
path: "token".to_string(),
}),
..Default::default()
}]),
}),
..Default::default()
},
empty_dir_volume("run"),
empty_dir_volume("logs"),
];
if !client_tls_secret_name.is_empty() {
environment.extend([
env_var("OPENSHELL_TLS_CA", "/var/run/secrets/openshell-tls/ca.crt"),
env_var(
"OPENSHELL_TLS_CERT",
"/var/run/secrets/openshell-tls/tls.crt",
),
env_var(
"OPENSHELL_TLS_KEY",
"/var/run/secrets/openshell-tls/tls.key",
),
]);
volume_mounts.push(volume_mount(
"client-tls",
"/var/run/secrets/openshell-tls",
true,
));
volumes.push(secret_volume("client-tls", client_tls_secret_name, None));
}
let mut command = vec![
"/openshell-supervisor".to_string(),
"--backend-descriptor-file".to_string(),
BACKEND_DESCRIPTOR_PATH.to_string(),
"--auth-bundle-file".to_string(),
SUPERVISOR_AUTH_BUNDLE_PATH.to_string(),
"--workdir".to_string(),
"/sandbox".to_string(),
"--health-socket-path".to_string(),
CONTROL_HEALTH_SOCKET_PATH.to_string(),
];
if let Some(url) = https_proxy {
command.extend(["--upstream-proxy".to_string(), url.to_string()]);
}
if let Some(hosts) = no_proxy {
command.extend(["--upstream-no-proxy".to_string(), hosts.to_string()]);
}
if proxy_auth_secret.is_some() {
command.extend([
"--upstream-proxy-auth-file".to_string(),
openshell_core::container_paths::UPSTREAM_PROXY_AUTH_MOUNT_PATH.to_string(),
]);
}
if proxy_auth_allow_insecure {
command.push("--upstream-proxy-auth-allow-insecure".to_string());
}
if proxy_connect_by_hostname {
command.push("--upstream-proxy-connect-by-hostname".to_string());
}
if let Some((secret_name, secret_key)) = proxy_auth_secret {
let auth_path = Path::new(openshell_core::container_paths::UPSTREAM_PROXY_AUTH_MOUNT_PATH);
let mount_path = auth_path
.parent()
.and_then(Path::to_str)
.ok_or_else(|| "upstream proxy authentication path has no UTF-8 parent".to_string())?;
let item_path = auth_path
.file_name()
.and_then(|name| name.to_str())
.ok_or_else(|| {
"upstream proxy authentication path has no UTF-8 file name".to_string()
})?;
volume_mounts.push(volume_mount("upstream-proxy-auth", mount_path, true));
volumes.push(secret_volume(
"upstream-proxy-auth",
secret_name,
Some(KeyToPath {
key: secret_key.to_string(),
path: item_path.to_string(),
..Default::default()
}),
));
}
if let Some(socket_path) = provider_spiffe_socket_path {
environment.push(env_var(
openshell_core::sandbox_env::PROVIDER_SPIFFE_WORKLOAD_API_SOCKET,
socket_path,
));
let mount_path = Path::new(socket_path)
.parent()
.and_then(Path::to_str)
.ok_or_else(|| "SPIFFE socket path has no UTF-8 parent".to_string())?;
volume_mounts.push(volume_mount("spiffe-workload-api", mount_path, true));
volumes.push(Volume {
name: "spiffe-workload-api".to_string(),
csi: Some(CSIVolumeSource {
driver: "csi.spiffe.io".to_string(),
read_only: Some(true),
..Default::default()
}),
..Default::default()
});
}
let mut container = Container {
name: "supervisor".to_string(),
image: Some(supervisor_image.to_string()),
command: Some(command),
termination_message_policy: Some("FallbackToLogsOnError".to_string()),
env: Some(environment),
readiness_probe: Some(Probe {
exec: Some(ExecAction {
command: Some(vec![
"/openshell-supervisor".to_string(),
"health".to_string(),
"--socket".to_string(),
CONTROL_HEALTH_SOCKET_PATH.to_string(),
]),
}),
period_seconds: Some(1),
failure_threshold: Some(3),
..Default::default()
}),
security_context: Some(SecurityContext {
run_as_user: Some(i64::from(control_uid)),
run_as_group: Some(i64::from(control_gid)),
run_as_non_root: Some(true),
read_only_root_filesystem: Some(true),
allow_privilege_escalation: Some(false),
capabilities: Some(Capabilities {
drop: Some(vec!["ALL".to_string()]),
..Default::default()
}),
..Default::default()
}),
volume_mounts: Some(volume_mounts),
..Default::default()
};
if let Some(policy) = supervisor_pull_policy {
container.image_pull_policy = Some(policy.as_kubernetes_str().to_string());
}
Ok(Pod {
metadata: ObjectMeta {
name: Some(names.supervisor_pod.clone()),
namespace: Some(namespace.to_string()),
owner_references: Some(vec![owner]),
labels: Some(labels),
annotations: Some(BTreeMap::from([(
"openshell.ai/sandbox-id".to_string(),
sandbox_id.to_string(),
)])),
..Default::default()
},
spec: Some(PodSpec {
service_account_name: Some(service_account_name.to_string()),
image_pull_secrets: Some(
image_pull_secrets
.iter()
.map(|name| LocalObjectReference { name: name.clone() })
.collect(),
),
automount_service_account_token: Some(false),
scheduling_gates: Some(vec![PodSchedulingGate {
name: "openshell.ai/bootstrap".to_string(),
}]),
security_context: Some(PodSecurityContext {
fs_group: Some(i64::from(control_gid)),
fs_group_change_policy: Some("OnRootMismatch".to_string()),
seccomp_profile: Some(k8s_openapi::api::core::v1::SeccompProfile {
type_: "RuntimeDefault".to_string(),
..Default::default()
}),
..Default::default()
}),
restart_policy: Some("Never".to_string()),
containers: vec![container],
volumes: Some(volumes),
..Default::default()
}),
..Default::default()
})
}
#[allow(clippy::too_many_arguments)]
#[must_use]
pub fn sandbox_bootstrap_secret(
namespace: &str,
names: &SandboxRuntimeNames,
sandbox_id: &str,
boundary_config: Vec<u8>,
boundary_certificate: Vec<u8>,
boundary_private_key: Vec<u8>,
owner: OwnerReference,
) -> Secret {
Secret {
metadata: ObjectMeta {
name: Some(names.sandbox_secret.clone()),
namespace: Some(namespace.to_string()),
owner_references: Some(vec![owner]),
labels: Some(common_labels(sandbox_id, SANDBOX_SECRET_COMPONENT)),
..Default::default()
},
data: Some(BTreeMap::from([
(BOUNDARY_CONFIG_KEY.to_string(), ByteString(boundary_config)),
(
BOUNDARY_CERTIFICATE_KEY.to_string(),
ByteString(boundary_certificate),
),
(
BOUNDARY_PRIVATE_KEY.to_string(),
ByteString(boundary_private_key),
),
])),
immutable: Some(true),
type_: Some("Opaque".to_string()),
..Default::default()
}
}
#[must_use]
#[allow(clippy::too_many_arguments)]
pub fn supervisor_bootstrap_secret(
namespace: &str,
names: &SandboxRuntimeNames,
sandbox_id: &str,
backend_descriptor: Vec<u8>,
supervisor_auth_bundle: Vec<u8>,
proxy_ca_certificate: Vec<u8>,
proxy_ca_private_key: Vec<u8>,
owner: OwnerReference,
) -> Secret {
Secret {
metadata: ObjectMeta {
name: Some(names.supervisor_secret.clone()),
namespace: Some(namespace.to_string()),
owner_references: Some(vec![owner]),
labels: Some(common_labels(sandbox_id, SUPERVISOR_SECRET_COMPONENT)),
..Default::default()
},
data: Some(BTreeMap::from([
(
BACKEND_DESCRIPTOR_KEY.to_string(),
ByteString(backend_descriptor),
),
(
SUPERVISOR_AUTH_BUNDLE_KEY.to_string(),
ByteString(supervisor_auth_bundle),
),
(
PROXY_CA_CERTIFICATE_KEY.to_string(),
ByteString(proxy_ca_certificate),
),
(
PROXY_CA_PRIVATE_KEY.to_string(),
ByteString(proxy_ca_private_key),
),
])),
immutable: Some(true),
type_: Some("Opaque".to_string()),
..Default::default()
}
}
#[must_use]
pub fn sandbox_owner_reference(
name: &str,
uid: &str,
api_version: &str,
controller: bool,
) -> OwnerReference {
OwnerReference {
api_version: api_version.to_string(),
kind: "Sandbox".to_string(),
name: name.to_string(),
uid: uid.to_string(),
controller: controller.then_some(true),
// The driver's RBAC intentionally does not permit mutating Sandbox
// finalizers. Kubernetes garbage collection does not require this bit,
// and setting it would make admission fail under
// OwnerReferencesPermissionEnforcement.
block_owner_deletion: Some(false),
}
}
fn pair_labels(sandbox_id: &str, role: &str) -> BTreeMap<String, String> {
BTreeMap::from([
(
BOUNDARY_PAIR_LABEL.to_string(),
pair_label_value(sandbox_id),
),
(BOUNDARY_ROLE_LABEL.to_string(), role.to_string()),
])
}
fn common_labels(sandbox_id: &str, component: &str) -> BTreeMap<String, String> {
BTreeMap::from([
(
"openshell.ai/managed-by".to_string(),
"openshell".to_string(),
),
(
"openshell.ai/sandbox-id".to_string(),
sandbox_id.to_string(),
),
("openshell.ai/component".to_string(), component.to_string()),
])
}
fn control_labels(sandbox_id: &str, gateway_id: &str) -> BTreeMap<String, String> {
let mut labels = common_labels(sandbox_id, "supervisor");
labels.extend(pair_labels(sandbox_id, "supervisor"));
labels.insert(
"openshell.ai/gateway-id".to_string(),
gateway_id.to_string(),
);
labels
}
fn env_var(name: &str, value: &str) -> EnvVar {
EnvVar {
name: name.to_string(),
value: Some(value.to_string()),
..Default::default()
}
}
fn volume_mount(name: &str, mount_path: &str, read_only: bool) -> VolumeMount {
VolumeMount {
name: name.to_string(),
mount_path: mount_path.to_string(),
read_only: read_only.then_some(true),
..Default::default()
}
}
fn secret_volume(name: &str, secret_name: &str, item: Option<KeyToPath>) -> Volume {
Volume {
name: name.to_string(),
secret: Some(SecretVolumeSource {
secret_name: Some(secret_name.to_string()),
default_mode: Some(0o440),
items: item.map(|item| vec![item]),
..Default::default()
}),
..Default::default()
}
}
fn empty_dir_volume(name: &str) -> Volume {
Volume {
name: name.to_string(),
empty_dir: Some(EmptyDirVolumeSource::default()),
..Default::default()
}
}
#[cfg(test)]
mod tests {
use super::*;
fn owner() -> OwnerReference {
sandbox_owner_reference("demo", "uid-1", "agents.x-k8s.io/v1beta1", false)
}
#[test]
fn service_selects_only_the_workload_boundary() {
let names = SandboxRuntimeNames::new("4b67c0d0-1111-2222-3333-444444444444");
let service = boundary_service("sandbox", &names, "pair", 5500, owner());
assert_eq!(
service.spec.unwrap().selector.unwrap()[BOUNDARY_ROLE_LABEL],
"workload"
);
}
#[test]
fn owner_reference_does_not_require_finalizer_mutation_permission() {
assert_eq!(owner().block_owner_deletion, Some(false));
}
#[test]
fn supervisor_pod_is_gated_non_restarting_and_unprivileged() {
let names = SandboxRuntimeNames::new("pair");
let pod = supervisor_pod(
"sandbox",
&names,
"pair",
"demo",
"gateway",
"supervisor:latest",
Some(crate::KubernetesImagePullPolicy::IfNotPresent),
"sandbox-sa",
1000,
1000,
&["registry-credentials".to_string()],
"https://gateway:8080",
"client-tls",
"{}",
"info",
600,
None,
None,
None,
false,
false,
None,
owner(),
)
.expect("render supervisor Pod");
let pod_spec = pod.spec.as_ref().expect("Pod spec");
assert_eq!(
pod.metadata
.owner_references
.as_ref()
.expect("owner references")[0]
.controller,
None
);
let container = &pod_spec.containers[0];
assert_eq!(container.image_pull_policy.as_deref(), Some("IfNotPresent"));
assert_eq!(pod_spec.automount_service_account_token, Some(false));
assert_eq!(pod_spec.restart_policy.as_deref(), Some("Never"));
assert_eq!(
pod_spec
.scheduling_gates
.as_ref()
.and_then(|gates| gates.first())
.map(|gate| gate.name.as_str()),
Some("openshell.ai/bootstrap")
);
assert_eq!(
pod_spec
.image_pull_secrets
.as_ref()
.expect("image pull secrets")[0]
.name
.as_str(),
"registry-credentials"
);
let pod_security = pod_spec
.security_context
.as_ref()
.expect("Pod security context");
assert_eq!(pod_security.fs_group, Some(1000));
assert_eq!(
pod_security
.seccomp_profile
.as_ref()
.map(|profile| profile.type_.as_str()),
Some("RuntimeDefault")
);
let container_security = container
.security_context
.as_ref()
.expect("container security context");
assert_eq!(container_security.run_as_user, Some(1000));
assert_eq!(container_security.run_as_non_root, Some(true));
assert_eq!(container_security.read_only_root_filesystem, Some(true));
assert_eq!(
container_security
.capabilities
.as_ref()
.and_then(|capabilities| capabilities.drop.as_ref()),
Some(&vec!["ALL".to_string()])
);
assert_eq!(
container
.readiness_probe
.as_ref()
.and_then(|probe| probe.exec.as_ref())
.and_then(|exec| exec.command.as_ref()),
Some(&vec![
"/openshell-supervisor".to_string(),
"health".to_string(),
"--socket".to_string(),
CONTROL_HEALTH_SOCKET_PATH.to_string(),
])
);
let command = container.command.as_ref().unwrap();
assert!(
command
.windows(2)
.any(|args| args == ["--health-socket-path", CONTROL_HEALTH_SOCKET_PATH])
);
let env = container.env.as_ref().unwrap();
let env_value = |name: &str| {
env.iter()
.find(|variable| variable.name == name)
.and_then(|variable| variable.value.as_deref())
};
assert_eq!(
env_value(openshell_core::sandbox_env::SSH_SOCKET_SHARED),
Some("true")
);
assert_eq!(
env_value(openshell_core::sandbox_env::PROXY_CA_CERT),
Some(PROXY_CA_CERTIFICATE_PATH)
);
assert_eq!(
env_value(openshell_core::sandbox_env::PROXY_CA_KEY),
Some(PROXY_CA_PRIVATE_KEY_PATH)
);
let mount = container
.volume_mounts
.as_ref()
.unwrap()
.iter()
.find(|mount| mount.name == "bootstrap")
.expect("durable supervisor material is mounted into supervisor");
assert_eq!(mount.mount_path, "/.openshell/supervisor");
assert_eq!(mount.read_only, Some(true));
}
#[test]
fn bootstrap_secrets_are_immutable_and_split_by_trust_domain() {
let names = SandboxRuntimeNames::new("pair");
let sandbox = sandbox_bootstrap_secret(
"sandbox",
&names,
"pair",
Vec::new(),
Vec::new(),
Vec::new(),
owner(),
);
assert_eq!(
sandbox.metadata.labels.as_ref().unwrap()["openshell.ai/component"],
SANDBOX_SECRET_COMPONENT
);
assert_eq!(sandbox.immutable, Some(true));
let sandbox_keys = sandbox
.data
.unwrap()
.into_keys()
.collect::<std::collections::BTreeSet<_>>();
assert_eq!(
sandbox_keys,
std::collections::BTreeSet::from([
BOUNDARY_CERTIFICATE_KEY.to_string(),
BOUNDARY_CONFIG_KEY.to_string(),
BOUNDARY_PRIVATE_KEY.to_string(),
])
);
let supervisor = supervisor_bootstrap_secret(
"sandbox",
&names,
"pair",
Vec::new(),
Vec::new(),
Vec::new(),
Vec::new(),
owner(),
);
assert_eq!(
supervisor.metadata.labels.as_ref().unwrap()["openshell.ai/component"],
SUPERVISOR_SECRET_COMPONENT
);
assert_eq!(supervisor.immutable, Some(true));
let supervisor_keys = supervisor
.data
.unwrap()
.into_keys()
.collect::<std::collections::BTreeSet<_>>();
assert_eq!(
supervisor_keys,
std::collections::BTreeSet::from([
PROXY_CA_CERTIFICATE_KEY.to_string(),
PROXY_CA_PRIVATE_KEY.to_string(),
SUPERVISOR_AUTH_BUNDLE_KEY.to_string(),
BACKEND_DESCRIPTOR_KEY.to_string(),
])
);
}
#[test]
fn generated_proxy_ca_material_is_pem_encoded() {
let material = generate_proxy_ca_material().unwrap();
assert!(
material
.certificate_pem
.starts_with("-----BEGIN CERTIFICATE-----")
);
assert!(material.private_key_pem.contains("PRIVATE KEY"));
}
}
crates/openshell-driver-mxc/src/driver.rs
+20 -11
View File
@@ -55,6 +55,9 @@ pub enum MxcBackend {
/// environment variables / CLI flags via the standard gateway precedence chain.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
// These are independent, user-facing feature switches in the flat gateway
// configuration schema rather than one compound state machine.
#[allow(clippy::struct_excessive_bools)]
pub struct MxcComputeConfig {
/// Path to `wxc-exec.exe`. Required for live runs.
pub wxc_exec_path: String,
@@ -858,7 +861,9 @@ async fn run_lifecycle(
} else {
None
};
let host_proxy_ca_paths = host_proxy.as_ref().and_then(|proxy| proxy.ca_file_paths());
let host_proxy_ca_paths = host_proxy
.as_ref()
.and_then(openshell_supervisor_network::host::HostProxyHandle::ca_file_paths);
drop(reserved_proxy_listener.take());
if let Some(addr) = proxy_addr {
{
@@ -1199,6 +1204,7 @@ fn make_sandbox_with_condition(
sandbox_fd: String::new(),
conditions: vec![condition.clone()],
deleting,
..Default::default()
}),
}
}
@@ -1519,10 +1525,12 @@ mod lifecycle_tests {
"-Command".into(),
format!("Set-Content -LiteralPath {hello} -Value hi"),
];
let mut config = MxcComputeConfig::default();
config.backend = MxcBackend::ProcessContainer;
config.egress_proxy = true;
config.egress_proxy_addr = "127.0.0.1:18080".into();
let config = MxcComputeConfig {
backend: MxcBackend::ProcessContainer,
egress_proxy: true,
egress_proxy_addr: "127.0.0.1:18080".into(),
..Default::default()
};
let backend = MxcComputeBackend::new_mocked(config);
let mut stream = backend.watch_sandboxes().await;
@@ -1539,7 +1547,6 @@ mod lifecycle_tests {
}],
binaries: vec![NetworkBinary {
path: "/usr/bin/curl".into(),
..Default::default()
}],
},
);
@@ -1570,7 +1577,7 @@ mod lifecycle_tests {
.as_u64()
.expect("proxy localhost port");
assert!(proxy_port > 0);
assert!(proxy_port <= u64::from(u16::MAX));
assert!(u16::try_from(proxy_port).is_ok());
assert!(
recorded["network"]["proxy"].get("host").is_none(),
"proxy must not contain 'host' key"
@@ -1610,7 +1617,7 @@ mod lifecycle_tests {
}
}
Ok(_) => break,
Err(_) => continue,
Err(_) => {}
}
}
assert!(saw_redirect, "expected EgressRedirect platform event");
@@ -1745,9 +1752,11 @@ mod lifecycle_tests {
#[tokio::test]
async fn governed_egress_rejects_network_middleware_before_lifecycle() {
let mut config = MxcComputeConfig::default();
config.egress_proxy = true;
config.egress_proxy_addr = "127.0.0.1:18080".into();
let config = MxcComputeConfig {
egress_proxy: true,
egress_proxy_addr: "127.0.0.1:18080".into(),
..Default::default()
};
let backend = MxcComputeBackend::new_mocked(config);
let mut policy = fs_policy(&[]);
crates/openshell-driver-mxc/tests/wxc_exec_real.rs
+20 -28
View File
@@ -272,7 +272,7 @@ fn dryrun_rejects_unknown_containment() {
}
/// The most important dry-run test: build a typed Windows policy, run
/// `split_policy` (proxy_redirect 127.0.0.1:18080, containment
/// `split_policy` (`proxy_redirect` 127.0.0.1:18080, containment
/// "processcontainer"), and verify the resulting config with `--dry-run`.
///
/// This proves that the mapper's emitted JSON is accepted by the real binary —
@@ -346,10 +346,7 @@ fn dryrun_accepts_split_policy_output() {
fn probe_processcontainer(wxc: &PathBuf) -> Result<(), String> {
// Abort early if the mock env var is set — a stale OPENSHELL_MXC_MOCK_WXC
// would silently turn this "real" run back into a mock run.
if std::env::var("OPENSHELL_MXC_MOCK_WXC")
.map(|v| v == "1")
.unwrap_or(false)
{
if std::env::var("OPENSHELL_MXC_MOCK_WXC").is_ok_and(|value| value == "1") {
return Err(
"OPENSHELL_MXC_MOCK_WXC=1 is set — unset it before running real enforcement tests"
.to_string(),
@@ -395,16 +392,17 @@ fn probe_processcontainer(wxc: &PathBuf) -> Result<(), String> {
|| combined.contains("not enabled")
{
// Extract the message if possible for a more useful skip reason.
let reason = if let Ok(v) =
let reason =
serde_json::from_str::<serde_json::Value>(&String::from_utf8_lossy(&out.stdout))
{
v["error"]["message"]
.as_str()
.unwrap_or("backend_error (E_NOTIMPL)")
.to_string()
} else {
"backend_error (velocity keys not enabled)".to_string()
};
.map_or_else(
|_| "backend_error (velocity keys not enabled)".to_string(),
|value| {
value["error"]["message"]
.as_str()
.unwrap_or("backend_error (E_NOTIMPL)")
.to_string()
},
);
return Err(reason);
}
@@ -421,7 +419,7 @@ fn probe_processcontainer(wxc: &PathBuf) -> Result<(), String> {
}
/// Probe the released binary's live `network.proxy` support separately from
/// ordinary ProcessContainer support. Some builds accept the proxy JSON during
/// ordinary `ProcessContainer` support. Some builds accept the proxy JSON during
/// `--dry-run` but return `ERROR_INVALID_PARAMETER` from the live launcher.
fn probe_processcontainer_proxy(wxc: &PathBuf) -> Result<(), String> {
let (_tempdir, temp_path) = temp_fixture();
@@ -472,15 +470,12 @@ fn probe_processcontainer_proxy(wxc: &PathBuf) -> Result<(), String> {
))
}
/// Probe the isolation_session backend.
/// Probe the `isolation_session` backend.
///
/// Attempts a `provision` phase. Returns `Ok(sandbox_id)` when live, or
/// `Err(reason)` when the backend is unavailable (caller prints SKIP).
fn probe_isolation_session(wxc: &PathBuf) -> Result<String, String> {
if std::env::var("OPENSHELL_MXC_MOCK_WXC")
.map(|v| v == "1")
.unwrap_or(false)
{
if std::env::var("OPENSHELL_MXC_MOCK_WXC").is_ok_and(|value| value == "1") {
return Err(
"OPENSHELL_MXC_MOCK_WXC=1 is set — unset it before running real enforcement tests"
.to_string(),
@@ -667,9 +662,9 @@ fn pc_oneshot_in_policy_write_succeeds() {
);
}
/// Run an HTTPS request through the real driver and ProcessContainer. The
/// Run an HTTPS request through the real driver and `ProcessContainer`. The
/// workload explicitly reads the injected bundle before curl uses it, proving
/// that the driver's internal TLS share is reachable from the AppContainer.
/// that the driver's internal TLS share is reachable from the `AppContainer`.
#[tokio::test]
#[ignore = "requires real wxc-exec and outbound HTTPS"]
async fn pc_https_egress_reads_injected_ca_bundle() {
@@ -742,10 +737,7 @@ async fn pc_https_egress_reads_injected_ca_bundle() {
access: "read-only".to_string(),
..Default::default()
}],
binaries: vec![NetworkBinary {
path: cmd_string,
..Default::default()
}],
binaries: vec![NetworkBinary { path: cmd_string }],
},
)]),
..Default::default()
@@ -820,7 +812,7 @@ async fn pc_https_egress_reads_injected_ca_bundle() {
}
/// Write to a path OUTSIDE the granted dir; assert exit non-zero and file absent.
/// This is the genuine OS default-deny proof — the AppContainer blocks the write
/// This is the genuine OS default-deny proof — the `AppContainer` blocks the write
/// without requiring any host ACL lockdown. The mock can only fake this.
#[test]
#[ignore = "requires real wxc-exec"]
@@ -890,7 +882,7 @@ fn pc_oneshot_out_of_policy_write_denied() {
// ── Isolation session enforcement tests ──────────────────────────────────────
/// Full isolation_session round trip: provision → start → exec → stop →
/// Full `isolation_session` round trip: provision → start → exec → stop →
/// deprovision. `deprovision` runs in a drop-guard even on panic so the
/// single-session backend is never left orphaned.
#[test]
crates/openshell-driver-podman/Cargo.toml
+4
View File
@@ -17,6 +17,10 @@ path = "src/main.rs"
[dependencies]
openshell-core = { path = "../openshell-core", default-features = false, features = ["driver-extraction"] }
openshell-otel = { path = "../openshell-otel" }
openshell-isolation-interface = { path = "../openshell-isolation-interface" }
openshell-sandbox-backend = { path = "../openshell-sandbox-backend" }
tar = "0.4"
uuid = { workspace = true }
tokio = { workspace = true }
tonic = { workspace = true, features = ["transport"] }
crates/openshell-driver-podman/NETWORKING.md
+47 -448
View File
@@ -1,463 +1,62 @@
# Rootless Podman Networking
# Podman Networking
Deep-dive into how networking works in the Podman compute driver when running
rootless with pasta as the network backend. Covers the external tooling
(Podman, Netavark, pasta, aardvark-dns), the three nested namespace layers, and
the complete data paths for SSH, outbound traffic, and supervisor-to-gateway
communication.
For the general Podman driver architecture, lifecycle, API surface, and driver
comparison, see [README.md](README.md).
## Component Stack
Podman's networking is composed of four independent projects:
| Component | Language | Role |
|---|---|---|
| Podman | Go | Container runtime; orchestrates network lifecycle. |
| Netavark | Rust | Network backend; creates interfaces, bridges, firewall rules. |
| aardvark-dns | Rust | Authoritative DNS server for container name resolution. |
| pasta, part of passt | C | User-mode networking; L2-to-L4 socket translation for rootless containers. |
The key split: rootful containers default to Netavark bridge networking with
real kernel interfaces, while rootless containers commonly use pasta user-mode
networking without needing host privileges.
## How Netavark Works
Netavark is invoked by Podman as an external binary. It reads a JSON network
configuration from STDIN and executes one of three commands:
- `netavark setup <netns-path>` creates interfaces, assigns IPs, and sets up
firewall rules for NAT and port-forwarding.
- `netavark teardown <netns-path>` reverses setup and removes interfaces and
firewall rules.
- `netavark create` takes a partial network config and completes it by
assigning subnets and gateways.
For rootful bridge networking:
1. Podman creates a network namespace for the container.
2. Podman invokes `netavark setup` with the network config JSON.
3. Netavark creates a bridge, such as `podman0`, if it does not exist. The
default subnet is `10.88.0.0/16`.
4. Netavark creates a veth pair. One end goes into the container's netns and
the other attaches to the bridge.
5. Netavark assigns an IP from the subnet to the container's veth interface.
6. Netavark configures iptables or nftables rules for masquerade and port
mappings.
7. Netavark starts aardvark-dns when DNS is enabled, listening on the bridge
gateway address.
Only the external supervisor has external network connectivity. The workload
container uses `network=none`; its loopback DNS relay and TCP socket mediation
reach the supervisor through a protected Unix socket, not a veth or proxy
environment variable.
```text
Host Kernel
|
+-- Bridge interface, such as "podman0"
| |
| +-- veth pair endpoint, host side, container 1
| +-- veth pair endpoint, host side, container 2
|
+-- Host physical interface, such as eth0
|
+-- NAT, iptables or nftables rules managed by Netavark
workload container supervisor container
agent -> sandbox -- private UDS / gRPC -> policy proxy -> host network -> destination
|
+-- authenticated gateway callback
```
Netavark also supports macvlan networks, where the container gets a
sub-interface of a physical host NIC with its own MAC address, and external
plugins via a documented JSON API.
## Outer network fence
## How Pasta Works
The driver creates the workload without networks, host aliases, published
ports, or added capabilities. It checks the Podman inspect response before
launch and restart. The sandbox installs seccomp mediation and Landlock before
executing the agent. It does not create network namespaces, configure nftables,
or require `CAP_NET_ADMIN`.
Unprivileged users cannot create network interfaces on the host. They cannot
create veth pairs, bridges, or iptables rules. Netavark's bridge approach
cannot work directly for rootless containers without an additional rootless
networking layer.
TCP opens, TCP byte streams, DNS requests/replies, and lifecycle operations
share the authenticated gRPC channel. DNS is resolved and authorized by the
supervisor. General UDP is unsupported.
Pasta, part of the `passt` project, operates in userspace and translates
between the container's L2 TAP interface and the host's L4 sockets. It requires
no capabilities or privileges.
## Supervisor callback network
```text
Container Network Namespace
|
+-- TAP device, such as "eth0"
| ^
| | L2 frames, Ethernet
| v
+-- pasta process, userspace
|
| Translation: L2 frames <-> L4 sockets
|
v
Host Network Stack, native TCP/UDP/ICMP sockets
```
The supervisor companion uses Podman's host network. Host-gateway aliases and
the upstream corporate proxy apply only to the supervisor. The gateway's SSH
tunnel uses the supervisor relay over its private Unix socket, so the driver
does not publish a supervisor port.
For an outbound TCP connection from a container:
Rootful Podman uses the configured bridge and its gateway address. Rootless
local callbacks require the existing pasta path; slirp4netns or unknown helpers
require an explicitly remote `grpc_endpoint`. On macOS, Podman Machine provides
the runtime and host-loopback forwarding.
1. The application calls `connect()` to an external address.
2. The kernel routes the packet through the default gateway to the TAP device.
3. Pasta reads the raw Ethernet frame from the TAP file descriptor.
4. Pasta parses L2/L3/L4 headers and identifies the TCP SYN.
5. Pasta opens a native TCP socket on the host and calls `connect()` to the
same destination.
6. When the host socket connects, pasta reflects the SYN-ACK back through the
TAP as an L2 frame.
7. For ongoing data transfer, pasta translates between TAP frames and the host
socket, coordinating TCP windows and acknowledgments between the two sides.
These runtime-managed network helpers are outside the workload trust boundary.
Sharing the workload's user namespace preserves volume UID/GID mapping; it
does not share the workload's PID, mount, or network namespaces.
Pasta does not maintain per-connection packet buffers. It reflects observed
sending windows and ACKs directly between peers. This is a thinner translation
layer than a full TCP/IP stack.
## Troubleshooting
### Built-in Services
Inspect both containers with the same sandbox-ID label, distinguishing
`openshell.io/isolation-role=sandbox` from
`openshell.io/isolation-role=supervisor`.
Pasta includes minimal network services so the container stack can
auto-configure:
- Sandbox fails its qualification probe: use its log to identify the denied
kernel/runtime primitive. Do not add capabilities or disable runtime seccomp.
- Sandbox cannot authenticate to supervisor: check the private channel volume,
matching user namespace mappings, and shared SELinux label.
- Supervisor cannot call back: inspect its configured gateway endpoint,
credentials, host network, and gateway callback listener.
- DNS or egress denied: inspect supervisor policy decisions. Do not add a
workload network, resolver bypass, or direct gateway route.
- Pair is not Ready: check the supervisor health socket and gateway session.
A running workload container alone does not establish readiness.
| Service | Purpose |
|---|---|
| ARP proxy | Resolves the gateway address to the host's MAC address. |
| DHCP server | Hands out a single IPv4 address, usually matching the host's upstream interface. |
| NDP proxy | Handles IPv6 neighbor discovery and SLAAC prefix advertisement. |
| DHCPv6 server | Hands out a single IPv6 address, usually matching the host's upstream interface. |
By default there is no NAT. Pasta copies the host's IP addresses into the
container namespace.
### Local Connection Bypass
For connections between the container and the host, pasta implements a local
bypass path:
- Packets with a local destination skip L2 translation.
- TCP uses `splice(2)`.
- UDP uses `recvmmsg(2)` and `sendmmsg(2)`.
### Port Forwarding
By default, pasta uses auto-detection. It scans `/proc/net/tcp` and
`/proc/net/tcp6` periodically and automatically forwards ports that are bound
and listening. Port forwarding is configurable through pasta options.
### Security Properties
Pasta is designed for rootless use:
- No dynamic memory allocation after startup.
- All capabilities dropped, except `CAP_NET_BIND_SERVICE` when granted.
- Restrictive seccomp profile.
- Detaches into its own user, mount, IPC, UTS, and PID namespaces.
- No external dependencies beyond libc.
### Inter-Container Limitation
Unlike bridge networking, pasta containers are isolated from each other by
default. No virtual bridge connects them. Communication requires port mappings
through the host, pods with a shared network namespace, or opting into rootless
Netavark bridge networking with `podman network create`.
## Three Nested Namespaces
The Podman compute driver creates three layers of network isolation:
```text
Namespace 1: Host
|
pasta manages port forwarding, such as 127.0.0.1:<ephemeral>
gateway listens on its configured bind address and port
|
Namespace 2: Rootless Podman network namespace, managed by pasta
|
Bridge "openshell", often 10.89.x.0/24
aardvark-dns for container name resolution
|
Container netns
supervisor, proxy, and relay client run here
|
Namespace 3: Inner sandbox netns, created by supervisor
|
veth pair, such as 10.200.0.1 <-> 10.200.0.2
nftables forces ordinary traffic through proxy
user workload runs here
```
Pasta bridges namespace 1 and 2. The veth pair bridges namespace 2 and 3. The
proxy at the boundary of namespace 2 and 3 enforces network policy.
### Layer 1 Pasta
At driver startup, the driver ensures a Podman bridge network exists:
```rust
client.ensure_network(&config.network_name).await?;
```
This creates a bridge network named `openshell` by default, with DNS enabled.
In rootless mode, this bridge can exist inside a user namespace managed by
pasta. The bridge IP range is not reliably routable from the host.
```text
Host
|
127.0.0.1:<ephemeral>, pasta binds this on the host
|
pasta process, translates L4 sockets <-> L2 TAP frames
|
rootless network namespace
|
Bridge "openshell", such as 10.89.1.0/24
|
+-- 10.89.1.1, bridge gateway and aardvark-dns
|
+-- veth to container netns
|
10.89.1.2, container IP
```
### Layer 2 Container Networking
The container spec configures:
- `nsmode: "bridge"` to use the Podman bridge network.
- `networks` to attach to the configured bridge, `openshell` by default.
- `portmappings` with `host_port: 0`, `container_port: 2222`, and `protocol:
"tcp"` to publish the SSH compatibility port on an ephemeral host port.
- `hostadd` entries for `host.containers.internal` and
`host.openshell.internal`, using Podman's `host-gateway` resolver or the
configured `host_gateway_ip`.
Pasta is not explicitly configured by the driver. The driver requests bridge
mode and logs the network backend that Podman reports at startup.
The `host.containers.internal` hostname is injected into `/etc/hosts` so the
supervisor can reach the gateway on the host. Linux defaults to
`host-gateway`; macOS Podman machine defaults to `192.168.127.254`, gvproxy's
host-loopback IP, because older Podman machine images can fail to resolve
`host-gateway`. Override this with `host_gateway_ip` or
`OPENSHELL_PODMAN_HOST_GATEWAY_IP` when a Podman machine uses a non-standard
host-loopback address.
If `OPENSHELL_GRPC_ENDPOINT` is empty, the driver auto-detects:
```rust
if config.grpc_endpoint.is_empty() {
let scheme = if config.tls_enabled() {
"https"
} else {
"http"
};
config.grpc_endpoint =
format!("{scheme}://host.containers.internal:{}", config.gateway_port);
}
```
The bridge gateway IP is not a stable substitute in rootless mode because it
can live inside the user namespace rather than on the host.
Before the gateway binds its serving sockets, the driver reports the callback
listener required by the selected topology:
- Rootful Linux Podman reports the configured bridge's gateway address exactly.
- Rootless Linux Podman explicitly reporting pasta requests the private IPv4
source address selected by the host's default route. This avoids guessing
among private interfaces on a multihomed host.
- Rootless Linux Podman reporting slirp4netns, another named helper, or no
helper cannot use a direct local callback listener. The driver fails startup
unless `grpc_endpoint` names an explicitly remote endpoint. Supporting
slirp4netns requires a relay inside Podman's rootless network namespace.
- Podman Machine requests IPv4 loopback because gvproxy terminates the host
forwarding path there.
- An explicitly remote callback endpoint requests no additional local listener.
On Linux, an explicit `host_gateway_ip` is reported exactly for rootful Podman
and rootless pasta because the driver maps both local callback aliases to that
literal. Other rootless helpers still fail closed. Podman Machine requests
gateway loopback because its configured address is guest-visible and gvproxy
terminates that route on host loopback. The gateway validates and binds every
accepted callback requirement. If the primary listener covers the requested
address, the gateway reuses it and relies on sandbox JWT authorization to limit
the supervisor's RPCs. Otherwise, it creates an additional listener that
exposes only the gateway's sandbox-callable gRPC methods. Operator, health,
reflection, and HTTP requests must use the primary listener.
### Layer 3 Inner Sandbox Network Namespace
Inside the container, the supervisor creates another network namespace for the
user workload:
```text
Container on the Podman bridge
|
Supervisor process, running in container's default netns
|
+-- Proxy listener at the inner namespace gateway address
|
+-- veth pair
|
+-- Inner network namespace
|
sandbox-side veth address
|
default route -> supervisor-side veth address
|
user code runs here
|
nftables rules:
ACCEPT -> proxy TCP
ACCEPT -> loopback
ACCEPT -> established/related
LOG -> TCP SYN bypass attempts
REJECT -> TCP
LOG -> UDP bypass attempts
REJECT -> UDP
```
The supervisor uses `nsenter --net=` rather than `ip netns exec` to avoid sysfs
remount issues that arise under rootless Podman where real host
`CAP_SYS_ADMIN` is unavailable.
For a policy with explicit `protocol: tcp` endpoints, this same inner namespace
also hosts policy DNS and transparent TCP capture. The supervisor answers only
policy-eligible names with epoch-scoped synthetic addresses, redirects TCP to
those synthetic ranges into its transparent listener, and leaves direct real-IP
dials subject to the terminal bypass fence. The Podman driver advertises this
substrate through its driver-owned runtime capability; sandbox image and policy
environment values cannot opt into it independently.
The container spec preserves Podman's resolver search domains and options.
Policy DNS captures both UDP and TCP in the inner namespace, so it does not
depend on libc honoring `use-vc` and does not change ordinary short-name
resolution for sandboxes that do not use native TCP.
A tmpfs is mounted at `/run/netns` in the container spec so the supervisor can
create named network namespaces. In rootless Podman this directory does not
exist on the host, so a private tmpfs gives the supervisor its own writable
`/run/netns` without needing host filesystem access.
## Complete Data Paths
### SSH Session
```text
Client, openshell CLI
|
1. gRPC: CreateSshSession -> gateway, returns token and connect_path
2. HTTP CONNECT /connect/ssh to gateway
headers: x-sandbox-id, x-sandbox-token
|
Gateway
|
3. Looks up SupervisorSession for sandbox_id
4. Sends RelayOpen{channel_id} over ConnectSupervisor bidi stream
|
gRPC traverses host -> pasta translation -> container bridge
|
Supervisor inside container
|
5. Receives RelayOpen, opens new RelayStream RPC back to gateway
6. Sends RelayInit{channel_id} on the stream
7. Connects to Unix socket /run/openshell/ssh.sock
8. Bidirectional bridge: RelayStream <-> Unix socket
|
SSH daemon inside container, Unix socket only
|
9. Authenticates. Access is gated by the relay chain.
10. Spawns shell process
11. Shell enters inner netns via setns(fd, CLONE_NEWNET)
|
User shell in sandbox netns
```
The SSH daemon listens on a Unix socket with restrictive permissions. The
published TCP port mapping exists in the container spec for compatibility and
health/debug paths. Normal SSH communication uses the gRPC reverse-connect relay
pattern.
### Outbound HTTP Request
```text
User code in inner netns
|
1. curl https://api.example.com
HTTP_PROXY points at the local sandbox proxy
|
2. TCP connect to proxy
allowed by nftables as the only ordinary egress destination
|
3. HTTP CONNECT api.example.com:443
|
Supervisor proxy in container netns
|
4. Policy evaluation with process identity
5. SSRF check
6. Optional L7 TLS intercept and HTTP method/path inspection
|
7. If allowed, TCP connect to api.example.com:443
from the container netns
|
8. Through Podman bridge -> pasta -> host -> internet
```
### Supervisor gRPC Callback
The Podman driver auto-detects the callback endpoint scheme based on whether
TLS client certificates are configured. When the RPM's auto-generated PKI is in
place, the endpoint is `https://host.containers.internal:17670` and the
supervisor connects with mTLS. Without TLS configuration, it falls back to
`http://host.containers.internal:<gateway-port>`.
```text
Supervisor in container netns
|
1. Connects to host.containers.internal:<port>
with mTLS when OPENSHELL_TLS_* paths are set
|
2. Routed through container default gateway
|
3. Pasta translates L2 frame -> host L4 socket when rootless backend uses pasta
|
4. Host TCP socket connects to gateway
|
Gateway
|
5. TLS handshake when enabled
6. ConnectSupervisor bidirectional stream established
7. Heartbeats at the interval accepted by the gateway
8. Reconnects with exponential backoff on failure
9. Same gRPC channel reused for RelayStream calls
```
The gateway binds to `127.0.0.1:17670` by default in the RPM packaging. Client
certificates are auto-generated by `openshell-gateway generate-certs` on first
start and bind-mounted into sandbox containers by the Podman driver.
## Differences from the Kubernetes Driver
| Aspect | Kubernetes | Podman, rootless pasta |
|---|---|---|
| Container or pod IP | Routable cluster-wide | Non-routable from the host in common rootless setups. |
| Network reachability | Pod IPs reachable from gateway | Bridge not reliably routable from host; requires host aliases or published ports. |
| Sandbox to gateway | Direct TCP to Kubernetes service or endpoint | `host.containers.internal` through bridge and rootless backend. |
| SSH transport | Reverse gRPC relay | Reverse gRPC relay. |
| Port publishing | Not needed for relay | Ephemeral host port remains in the container spec for compatibility and debug paths. |
| TLS | mTLS via Kubernetes secrets | mTLS via mounted client files, RPM defaults, or explicit configuration. |
| DNS | Kubernetes CoreDNS | Podman bridge DNS through aardvark-dns when DNS is enabled. |
| Network policy | Kubernetes network policy for pod ingress plus supervisor policy | nftables inside inner sandbox netns plus supervisor policy. |
| Supervisor delivery | Kubernetes driver managed pod image or template | OCI image volume mount. |
| Secrets | Kubernetes Secret volume and env vars | Per-sandbox JWT via Podman secret; TLS client materials from configured host files. |
Both drivers use the same reverse gRPC relay for SSH transport. The most
important Podman-specific difference is network reachability: in rootless
Podman, the bridge network is not reliably routable from the host, so
host-to-container and container-to-host communication must use host aliases,
published ports, or the supervisor relay.
## Port Assignments
| Port | Component | Purpose |
|---|---|---|
| `17670` | Gateway | Default local gRPC and HTTP multiplexed server port. |
| `2222` | Sandbox | Container port mapping default for the SSH compatibility port. |
| `3128` | Sandbox proxy | HTTP CONNECT proxy inside the sandbox network model. |
| `0` | Host | Ephemeral host port requested for the container SSH compatibility port. |
See the [driver overview](README.md) and
[Podman runtime documentation](https://docs.podman.io/en/latest/markdown/podman-run.1.html)
for runtime options.
crates/openshell-driver-podman/README.md
+95 -462
View File
@@ -1,489 +1,122 @@
# openshell-driver-podman
The Podman compute driver manages sandbox containers via the Podman REST API
over a Unix socket. It targets single-machine and developer environments where
rootless container isolation is preferred over a full Kubernetes cluster. The
driver runs in-process within the gateway server and delegates all sandbox
isolation enforcement to the `openshell-sandbox` supervisor binary, which is
sideloaded into each container via an OCI image volume mount.
The Podman compute driver runs inside the gateway and uses the native libpod
REST API over a Unix socket. Each sandbox has two independent containers:
When the gateway configures `[openshell.gateway.otlp]`, Podman compute-driver
spans export to the same OTLP/gRPC collector with the service name
`openshell-driver-podman`. The driver preserves the gateway trace context and
uses the same compute-driver RPC span names in its in-process and standalone
forms.
- `openshell-sandbox` owns the agent process in the workload container.
- `openshell-supervisor` evaluates policy, holds gateway credentials, and
proxies approved egress in a separate companion container.
`mise run gateway:podman` enables this export only when a local collector is
listening on `127.0.0.1:4317`. Otherwise, it omits the gateway OTLP configuration
so the development gateway does not repeatedly report export failures.
The driver provisions placement, identity, credentials, transport, and lifecycle.
The shared isolation interface supplies exec, attach, signal, terminate, binary
identity, DNS, TCP, and loopback-forwarding semantics.
Before creating the container, the driver inspects the final sandbox image and
captures its immutable image ID and raw OCI `Config.User`. Container creation
uses that image ID with pulling disabled, preventing a mutable tag from changing
between inspection and launch. The supervisor runs as root, resolves omitted
policy identity fields from the image declaration, and drops only agent
children to the completed identity. Named OCI components remain names after
validation; a missing group is filled with the user's numeric primary GID. Explicit
`process.run_as_user` and `process.run_as_group` values take precedence
independently.
## Runtime posture
For a rootless networking deep dive, see [NETWORKING.md](NETWORKING.md).
## Stop and Start
Stop stops the managed container without deleting it. The per-sandbox named
workspace volume, token and proxy-auth secrets, labels, and container metadata
remain intact. Start starts the same container and reuses the same named
volume. Stopped managed containers remain visible through list and watch
reconciliation. Delete remains responsible for removing the container,
driver-owned secrets, and workspace volume.
The stop call waits until Podman reports the container as stopped or exited.
This keeps an immediate start from racing a rootless Podman stop that is still
finishing after its API request returns.
Graceful gateway shutdown sends `StopSandbox` for each sandbox whose persisted
phase requires running compute without changing that persisted intent. On
startup, the gateway sends an idempotent `StartSandbox` request for the same
sandboxes, restarting their retained containers. Explicitly stopped sandboxes
remain excluded.
## Architecture
The Podman driver communicates with the Podman daemon over a Unix socket and
delegates sandbox isolation to the supervisor binary running inside each
container.
```mermaid
graph TB
CLI["openshell CLI"] -->|gRPC| GW["Gateway Server<br/>(openshell-server)"]
GW -->|in-process| PD["PodmanComputeDriver"]
PD -->|HTTP/1.1<br/>Unix socket| PA["Podman API"]
PA -->|OCI runtime<br/>crun/runc| C["Sandbox Container"]
C -->|image volume<br/>read-only| SV["Supervisor Binary<br/>/opt/openshell/bin/openshell-sandbox"]
SV -->|creates| NS["Nested Network Namespace<br/>veth pair + proxy"]
SV -->|enforces| LL["Landlock + seccomp"]
SV -->|gRPC callback| GW
```
## Isolation Model
The Podman driver provides the same protection layers as the other compute
drivers. The driver itself does not implement isolation primitives directly. It
configures the container so that the `openshell-sandbox` supervisor can enforce
them at runtime.
### Container Security Configuration
The container spec in `container.rs` sets these security-critical fields:
| Setting | Value | Rationale |
| Property | Workload | Supervisor |
|---|---|---|
| `user` | `0:0` | The supervisor needs root inside the container for namespace creation, proxy setup, Landlock, seccomp, and filesystem preparation. |
| `cap_drop` | Selected unneeded defaults | Podman's default capability set is already restricted. The driver drops capabilities the supervisor does not need. |
| `cap_add` | `SYS_ADMIN`, `NET_ADMIN`, `SYS_PTRACE`, `SYSLOG`, `DAC_READ_SEARCH`, `SETPCAP`, `KILL` | Grants supervisor-only capabilities required for namespace setup, process identity, bypass diagnostics, child bounding-set cleanup, and forwarding shutdown signals to a workload that runs as the sandbox user. Policy DNS binds an unprivileged supervisor port and does not require `NET_BIND_SERVICE`. |
| `no_new_privileges` | `true` | Prevents privilege escalation after exec. |
| `seccomp_profile_path` | `unconfined` | The supervisor installs its own policy-aware BPF filter. A container-level profile can block Landlock/seccomp syscalls during setup. |
| `mounts` | Private tmpfs at `/run/netns` | Lets the supervisor create named network namespaces in rootless Podman. |
| CDI GPU devices | Opaque `driver_config.cdi_devices` values when set, otherwise the requested count of NVIDIA CDI GPUs selected in round-robin order. Local `/dev/dxg` permits `nvidia.com/gpu=all` as a WSL2 all-only compatibility fallback, where it counts as one selectable device. | Exposes requested GPUs to GPU-enabled sandbox containers. Exact CDI device lists must not contain duplicates and must match the effective GPU count. |
| UID/GID | Pinned non-root workload identity | Same mapped identity |
| Capabilities | Drop all; add none | Drop all; add none |
| Seccomp | Runtime default plus sandbox-installed filters | Runtime default |
| Network | `none`; loopback only | Podman host network |
| Gateway JWT and upstream credentials | Never mounted | Podman secrets |
| User volumes and CDI devices | Workload only | Never mounted |
| Channel | Private named volume, writable | Same volume, read-only |
The restricted agent child does not retain these supervisor privileges.
Podman creates the namespaces and volume ownership before the workload runs.
Rootless operation uses the operator's Podman service and subordinate-ID
configuration; it does not require adding capabilities to either container.
The supervisor joins the workload's **user namespace only** to preserve UID/GID
mapping for shared-volume access. PID, mount, and network namespaces remain
separate. The channel volume uses shared SELinux relabeling (`:z`).
## Driver Config Mounts
Before starting either container, the driver uploads volume-relative archives
directly to the channel and workspace volume destinations. A rootfs upload on a
stopped Podman container does not populate nested named volumes. Restart restores
only the channel bootstrap into the existing channel volume, preserving the
workspace. The workload starts before the supervisor so its user namespace exists
when the supervisor joins it; a stopped supervisor resolves that namespace again
on its next start.
The gateway forwards the `podman` block from `--driver-config-json` to this
driver. The driver accepts user-supplied `mounts` entries with these Podman
mount types:
The runtime must pass the sandbox's unprivileged enforcement probe, including
nested seccomp notification and Landlock. Unsupported runtime defaults fail
closed; do not switch to an unconfined profile or add capabilities.
- `bind`: mounts an absolute host path when `[openshell.drivers.podman]`
has `enable_bind_mounts = true`.
- `volume`: mounts an existing Podman named volume. The driver validates that
the volume exists before provisioning and never creates or removes it. Podman
local-driver volumes created with bind options are treated as host bind
mounts and require `enable_bind_mounts = true`.
- `tmpfs`: mounts an in-memory filesystem with optional `options`,
`size_bytes`, and `mode`.
- `image`: mounts an OCI image through Podman's image-volume API. The driver
pulls the image during provisioning using the sandbox image pull policy.
## Protected channel and network enforcement
Host bind mounts are disabled by default because they expose gateway host paths
to sandbox requests. The driver still uses internal bind mounts for configured
TLS material; per-sandbox gateway JWTs are delivered through Podman secrets.
Podman `bind` mounts accept `source`, `target`, optional `read_only`, and an
optional `selinux_label` of `shared` (applies `:z`) or `private` (applies
`:Z`) for SELinux-enforcing hosts. User-supplied bind and volume mounts are
read-only by default; set `read_only: false` to make them writable. Podman
image and volume mounts do not support `subpath` in OpenShell driver config.
Mount `source` and `target` values must not contain surrounding whitespace.
Mount targets must be absolute container paths and must not replace
the workspace root (`/sandbox`) or overlap OpenShell supervisor files,
`/etc/openshell`, `/etc/openshell-tls`, or `/run/netns`.
Example named-volume usage:
```shell
podman volume create openshell-work
openshell sandbox create \
--driver-config-json '{"podman":{"mounts":[{"type":"volume","source":"openshell-work","target":"/sandbox/work"}]}}' \
-- claude
```text
agent -> openshell-sandbox === authenticated gRPC / private UDS === supervisor -> network
network=none TCP, DNS, control streams policy + gateway JWT
```
### Capability Breakdown
The workload has no external interface or published port. Seccomp socket
mediation carries TCP and DNS through one authenticated gRPC connection.
DNS remains supervisor-mediated; general UDP is unsupported. The driver sets
`net.ipv4.ip_unprivileged_port_start=0` in the isolated workload network
namespace so the sandbox's loopback DNS relay can bind port 53 without a
capability. No nftables or nested network namespace setup runs in the sandbox.
| Capability | Purpose |
|---|---|
| `SYS_ADMIN` | seccomp filter installation, namespace creation, and Landlock setup. |
| `NET_ADMIN` | Network namespace veth setup, IP address assignment, routes, and nftables. |
| `SYS_PTRACE` | Reading `/proc/<pid>/exe` and walking process ancestry for binary identity. |
| `SYSLOG` | Reading `/dev/kmsg` for bypass-detection diagnostics. |
| `DAC_READ_SEARCH` | Reading `/proc/<pid>/fd/` across UIDs so the proxy can resolve the binary responsible for a connection. |
| `SETPCAP` | Clearing the restricted child process capability bounding set before exec. |
The channel contains the sandbox bootstrap and sandbox-side TLS identity only.
Supervisor private keys and the runtime descriptor stay in the supervisor's private filesystem.
Landlock denies agent access to the top-level `/.openshell` control hierarchy.
The driver verifies Podman's reported `network=none` fence before launch and
restart. `host.containers.internal` and callback networking apply to the
supervisor, not the agent.
The driver intentionally keeps Podman's default `SETUID`, `SETGID`, `CHOWN`,
and `FOWNER` capabilities because the supervisor needs them to drop privileges
and prepare writable sandbox directories. It also keeps `SETPCAP` until child
setup so `drop_privileges()` can clear the child capability bounding set before
exec. It drops unneeded defaults such as
`DAC_OVERRIDE`, `FSETID`, `KILL`, `NET_RAW`, `SETFCAP`,
and `SYS_CHROOT`.
Gateway callbacks use the existing sandbox JWT and optional configured mTLS
bundle. The sandbox/supervisor channel always uses its separate, per-sandbox
mutual TLS material. These are distinct authentication relationships.
## Supervisor Sideloading
## Identity and trusted binaries
The supervisor binary is delivered to sandbox containers via Podman's OCI image
volume mechanism, distinct from both the Kubernetes pod-volume approach and the
VM's embedded guest bundle.
Both workload and supervisor images are pinned by immutable image ID. The
driver reads account files from a stopped workload-image container; it never
executes the image to resolve an account. Policy identity fields override OCI
`USER` independently. Named users/groups resolve against that image, including
supplementary groups. Root and unresolved identities fail before provisioning.
Images must not prepopulate the reserved `/.openshell` hierarchy; this prevents
image-controlled symlinks from aliasing private control state into user mounts.
```mermaid
sequenceDiagram
participant D as PodmanComputeDriver
participant P as Podman API
participant C as Sandbox Container
`sandbox_runtime_image` supplies the statically linked musl
`/openshell-sandbox` binary. Podman's read-only image volume delivers it to the
workload; user-namespace modes that cannot use image volumes retain the trusted
binary extraction path. `supervisor_image` supplies the dynamically linked
glibc `/openshell-supervisor` binary outside the workload. Image and request
environment belong to agent children, never the supervisor process.
D->>P: pull_image(supervisor, "missing")
D->>P: create_container(spec with image_volumes)
Note over P: Podman resolves image_volumes at<br/>libpod layer before OCI spec generation
P->>C: Mount supervisor image at /opt/openshell/bin (read-only)
D->>P: start_container
C->>C: entrypoint: /opt/openshell/bin/openshell-sandbox
```
## Lifecycle and readiness
The supervisor image from `deploy/docker/Dockerfile.supervisor` provides the
static `openshell-sandbox` binary at `/openshell-sandbox`.
Mounting that image at `/opt/openshell/bin` makes the binary available as
`/opt/openshell/bin/openshell-sandbox`.
Create builds both stopped containers and stages the private archives before
starting either container. The sandbox does not execute the agent until the
supervisor authenticates and confirms the common boundary contract. Failed
creation removes only containers created by that attempt, then cleans up
driver-owned volumes and secrets.
The container spec sets that binary as the entrypoint. This avoids relying on
the sandbox image entrypoint or command, which might otherwise append the
supervisor path as an argument to an image-provided shell.
Stop retains both containers, workspace, channel, and secrets. Start restores
the consumed sandbox bootstrap from a copy in the supervisor's private
filesystem, verifies the fence, and starts the same pair. A failed supervisor
start stops the workload. Delete removes the companion first, then the workload,
channel, workspace, and driver-owned secrets. User-owned volumes are retained.
## TLS
Only workload containers appear in sandbox list/watch results. Readiness uses
the supervisor's private health socket; there is no shell, legacy marker, or
TCP-listener shortcut. Watch reconciliation and supervisor exit/removal events
stop a running workload whose companion is unavailable. The gateway also
requires the authenticated supervisor session before publishing Ready.
When all three Podman TLS paths are set, the driver treats sandbox callbacks as
mTLS callbacks:
## Mounts, GPUs, and configuration
- `OPENSHELL_PODMAN_TLS_CA`
- `OPENSHELL_PODMAN_TLS_CERT`
- `OPENSHELL_PODMAN_TLS_KEY`
User `bind`, `volume`, `tmpfs`, and `image` mounts and CDI GPU selection remain
native Podman features and apply only to the workload. Bind mounts require the
operator's `enable_bind_mounts` opt-in. Reserved control paths and the workspace
root cannot be replaced. User-owned volumes are never created or deleted.
The driver validates that the TLS paths are provided as a complete set. Partial
configuration fails early instead of silently falling back to plaintext.
See [gateway configuration](../../docs/reference/gateway-config.mdx) for
operator settings and [NETWORKING.md](NETWORKING.md) for callback networking.
The supervisor uses Podman's host network and owns the upstream proxy settings.
Omit `health_check_interval_secs` to disable Podman's periodic health command.
Explicit zero is invalid. OpenShell still gates readiness on the supervisor's
authenticated health signal.
When enabled, the driver:
1. Switches the auto-detected endpoint scheme from `http://` to `https://`.
2. Bind-mounts the client cert files read-only into the container at
`/etc/openshell/tls/client/`.
3. Sets `OPENSHELL_TLS_CA`, `OPENSHELL_TLS_CERT`, and `OPENSHELL_TLS_KEY` to
the container-side paths.
The supervisor reads these env vars and uses them to establish an mTLS
connection back to the gateway. On SELinux systems, the bind mounts include
Podman's shared relabel option so the container process can read the files.
The RPM packaging auto-generates a self-signed PKI on first start via
`openshell-gateway generate-certs`. Client certs are placed in the CLI
auto-discovery directory (`~/.config/openshell/gateways/openshell/mtls/`) so
the CLI connects with mTLS without manual configuration. See
`deploy/rpm/CONFIGURATION.md` for the full RPM configuration reference.
## Network Model
Sandbox network isolation uses a two-layer approach: a Podman bridge network
for container-to-host communication, and a nested network namespace created by
the supervisor for sandbox process isolation.
```mermaid
graph TB
subgraph Host
GW["Gateway Server<br/>127.0.0.1:17670"]
PS["Podman Socket"]
end
subgraph Bridge["Podman Bridge Network (10.89.x.x)"]
subgraph Container["Sandbox Container"]
SV["Supervisor<br/>(root in user ns)"]
subgraph NestedNS["Nested Network Namespace"]
SP["Sandbox Process<br/>(resolved non-root identity)"]
VE2["veth1: 10.200.0.2"]
end
VE1["veth0: 10.200.0.1<br/>(CONNECT proxy)"]
SV --- VE1
VE1 ---|veth pair| VE2
end
end
GW -.->|SSH via supervisor relay<br/>gRPC session| SV
SV -->|gRPC callback via<br/>host.containers.internal| GW
SP -->|all egress via proxy| VE1
```
Key points:
- Bridge network: created by `client.ensure_network()` with DNS enabled.
Containers on the bridge can see each other at L3, but sandbox processes
cannot because they are isolated inside the nested netns.
- Nested netns: the supervisor creates a private `NetworkNamespace` with a veth
pair. Sandbox processes enter this netns via `setns(fd, CLONE_NEWNET)` in the
`pre_exec` hook, forcing ordinary traffic through the CONNECT proxy.
- Policy DNS and transparent TCP: the driver advertises the complete
`policy-dns-transparent-tcp` substrate. For explicit `protocol: tcp`
endpoints, the supervisor installs namespace-local DNS listeners, synthetic
routes, and TCP redirect rules before starting the workload. The container
disables Podman's implicit DNS search suffix so policy DNS evaluates the
exact endpoint name requested by the workload, and asks libc to use the
policy DNS TCP listener to avoid rootless Podman's nested UDP NAT return
path.
- Port publishing: the container spec still requests `host_port: 0` for the
configured SSH port. The gateway SSH tunnel uses the supervisor relay rather
than connecting directly to the published port.
- Host gateway: `host.containers.internal` and `host.openshell.internal` are
injected into `/etc/hosts` so containers can reach services on the gateway
host. Linux defaults to Podman's `host-gateway` resolver. macOS Podman
machine defaults to gvproxy's host-loopback IP, `192.168.127.254`, because
stale Podman machines may fail to resolve `host-gateway`.
- nsenter: the supervisor uses `nsenter --net=` instead of `ip netns exec` for
namespace operations, avoiding the sysfs remount path that fails in rootless
containers.
See [NETWORKING.md](NETWORKING.md) for the rootless Podman networking deep dive.
## Supervisor Relay
Podman follows the same end-to-end contract as the Kubernetes and VM drivers
for the in-container SSH relay: gateway config to `PodmanComputeConfig` to
sandbox environment to supervisor session registration on that path.
1. `[openshell.drivers.podman].ssh_socket_path` is deserialized into
`PodmanComputeConfig::ssh_socket_path` when the gateway builds the in-process
driver. The field defaults to `/run/openshell/ssh.sock` when omitted.
2. `build_env()` in `container.rs` sets `OPENSHELL_SSH_SOCKET_PATH` to that
value, alongside required vars such as `OPENSHELL_ENDPOINT` and
`OPENSHELL_SANDBOX_ID`. These driver-controlled entries overwrite template
environment variables to prevent spoofing.
3. The supervisor reads `OPENSHELL_SSH_SOCKET_PATH` and uses it for the Unix
socket the gateway's SSH stack bridges to.
The standalone `openshell-driver-podman` binary sets the same struct field from
`OPENSHELL_SANDBOX_SSH_SOCKET_PATH`.
## Credential Injection
Sandboxes authenticate to the gateway via mTLS using client materials bind-
mounted into the container from a Podman secret. No shared per-request secret
is injected as an environment variable.
| Credential | Mechanism | Visible in `inspect`? | Visible in `/proc/<pid>/environ`? |
|---|---|---|---|
| mTLS client cert/key | Bind-mounted file paths (`OPENSHELL_TLS_*` env vars point at them) | Yes (paths only) | Yes (paths only) |
| Sandbox identity | Plaintext env var | Yes | Yes |
| gRPC endpoint | Plaintext env var, override-protected | Yes | Yes |
| Supervisor relay socket path | Plaintext env var, override-protected | Yes | Yes |
The `build_env()` function inserts user-supplied variables first, then
unconditionally overwrites all security-critical variables to prevent spoofing
via sandbox templates:
- `OPENSHELL_SANDBOX`
- `OPENSHELL_SANDBOX_ID`
- `OPENSHELL_ENDPOINT`
- `OPENSHELL_SSH_SOCKET_PATH`
- `OPENSHELL_CONTAINER_IMAGE`
- `OPENSHELL_MAIN_PROCESS_SPEC`
## Sandbox Lifecycle
### Creation Flow
```mermaid
sequenceDiagram
participant GW as Gateway
participant D as PodmanComputeDriver
participant P as Podman API
GW->>D: create_sandbox(DriverSandbox)
D->>D: validate name + id
D->>D: validated_container_name()
D->>P: pull_image(supervisor, "missing")
D->>P: pull_image(sandbox_image, policy)
D->>P: create_volume(workspace)
Note over D: On failure below, rollback volume
D->>P: create_container(spec)
alt Conflict (409)
D->>P: remove_volume
D-->>GW: AlreadyExists
end
Note over D: On failure below, rollback container + volume
D->>P: start_container
D-->>GW: Ok
```
Each step rolls back previously-created resources on failure. The Conflict path
cleans up the volume because it is keyed by the new sandbox's ID, not the
conflicting container's ID.
### Readiness and Health
The container `healthconfig` marks the sandbox healthy when any of these
signals succeeds:
- Legacy marker file `/var/run/openshell-ssh-ready`.
- `test -S` on the configured supervisor Unix socket path.
- The prior TCP check for a listener on the in-container SSH port.
The Unix socket check allows relay-only backend readiness when the supervisor
exposes the socket without the old marker or published-port signal. Omitting
`health_check_interval_secs` disables these Podman/conmon probes, but it does
not bypass public readiness gating: the gateway keeps a backend-ready sandbox
in `Provisioning` with `SupervisorNotConnected` until its supervisor control
session is connected.
### Deletion Flow
1. Validate `sandbox_name` and stable `sandbox_id` from `DeleteSandboxRequest`.
2. Best-effort inspect cross-checks the container label when present, but
cleanup remains keyed by the request `sandbox_id`.
3. Best-effort stop, ignoring the stop result.
4. Force-remove the container.
5. Remove workspace volume derived from the request `sandbox_id`, warning on
failure and continuing.
If the container is already gone during inspect or remove, the driver still
performs idempotent volume cleanup using the request `sandbox_id` and
returns `Ok(false)` for the container-delete result. This prevents leaked
Podman resources after out-of-band container removal or label drift.
## Configuration
| Environment Variable | CLI Flag | Default | Description |
|---|---|---|---|
| `OPENSHELL_PODMAN_SOCKET` | `--podman-socket` | Probes known local Podman API sockets and uses the first responsive socket, then falls back to asking the `podman` CLI for the host-side socket. Fails to start if neither finds one. | Podman API Unix socket path. |
| `OPENSHELL_SANDBOX_IMAGE` | `--sandbox-image` | From gateway config | Default OCI image for sandboxes. |
| `OPENSHELL_SANDBOX_IMAGE_PULL_POLICY` | `--sandbox-image-pull-policy` | `if_not_present` | Pull policy: `always`, `if_not_present`, `never`, or `newer`. |
| `OPENSHELL_GRPC_ENDPOINT` | `--grpc-endpoint` | Auto-detected via `host.containers.internal` | Gateway gRPC endpoint for sandbox callbacks. |
| `OPENSHELL_GATEWAY_PORT` | `--gateway-port` | `17670` | Gateway port used for endpoint auto-detection by the standalone binary. |
| `OPENSHELL_NETWORK_NAME` | `--network-name` | `openshell` | Podman bridge network name. |
| `OPENSHELL_PODMAN_HOST_GATEWAY_IP` | `--host-gateway-ip` | empty on Linux, `192.168.127.254` on macOS | Host gateway IP used for sandbox host aliases. Empty uses Podman's `host-gateway` resolver. |
| `OPENSHELL_SANDBOX_SSH_SOCKET_PATH` | `--sandbox-ssh-socket-path` | `/run/openshell/ssh.sock` | Supervisor Unix socket path in `PodmanComputeConfig`. |
| `OPENSHELL_STOP_TIMEOUT` | `--stop-timeout` | `45` | Container stop timeout in seconds. |
| `OPENSHELL_SANDBOX_PIDS_LIMIT` | `--sandbox-pids-limit` | `2048` | Podman cgroup PID limit for sandbox containers. Omission uses OpenShell's `2048` default; explicit `0` is invalid. |
| `OPENSHELL_SUPERVISOR_IMAGE` | `--supervisor-image` | `ghcr.io/nvidia/openshell/supervisor:latest` through the gateway, required standalone | OCI image containing the supervisor binary. |
| `OPENSHELL_PODMAN_TLS_CA` | `--podman-tls-ca` | unset | Host path to the CA certificate mounted for sandbox mTLS. |
| `OPENSHELL_PODMAN_TLS_CERT` | `--podman-tls-cert` | unset | Host path to the client certificate mounted for sandbox mTLS. |
| `OPENSHELL_PODMAN_TLS_KEY` | `--podman-tls-key` | unset | Host path to the client private key mounted for sandbox mTLS. |
| `OPENSHELL_SANDBOX_HTTPS_PROXY` | `--sandbox-https-proxy` | unset | Corporate forward proxy URL for the supervisor's upstream TLS dials, chained with HTTP CONNECT. Credential-free `http://host:port` and `https://host:port` URLs are supported (scheme and port required). For an `https://` proxy the supervisor TLS-wraps the proxy connection, verifying the proxy certificate against the built-in and system roots plus `--sandbox-proxy-ca-bundle`. Plain-HTTP requests always dial directly. |
| `OPENSHELL_SANDBOX_NO_PROXY` | `--sandbox-no-proxy` | unset | Comma-separated `NO_PROXY` list (hostnames, domain suffixes, IPs, CIDRs, each with an optional `:port` qualifier) dialed directly instead of through the corporate proxy. IP/CIDR entries also match hostnames through their validated DNS resolution. |
| `OPENSHELL_SANDBOX_PROXY_AUTH_FILE` | `--sandbox-proxy-auth-file` | unset | Path to a file containing the proxy credentials as `user:pass`. Staged as a root-only Podman secret so credentials never appear in config or container metadata. Requires the insecure-auth acknowledgement below. |
| `OPENSHELL_SANDBOX_PROXY_AUTH_ALLOW_INSECURE` | `--sandbox-proxy-auth-allow-insecure` | unset | Explicit acknowledgement (`true`) that the credential is sent as cleartext Basic auth over the plain-TCP connection to the `http://` proxy. Required when the auth file is set with an `http://` proxy; not required for `https://` proxies (the credential travels inside the verified TLS session) but tolerated if set. Rejected when no auth file is configured. |
| `OPENSHELL_SANDBOX_PROXY_CONNECT_BY_HOSTNAME` | `--sandbox-proxy-connect-by-hostname` | unset | Send the destination hostname in CONNECT requests instead of a validated IP. Last resort for proxies whose ACLs filter on hostnames: the proxy then resolves the name itself, so sandbox SSRF/`allowed_ips` validation no longer binds the connection. |
| `OPENSHELL_PODMAN_USERNS` | `--userns` | unset | User namespace mode for sandbox containers (e.g. `auto`). When unset, containers use the default user namespace. |
| `OPENSHELL_SANDBOX_PROXY_CA_BUNDLE` | `--sandbox-proxy-ca-bundle` | unset | Path (on the gateway host) to a PEM CA bundle trusted for the corporate proxy. Bind-mounted read-only into the sandbox (a CA certificate is not secret). Trusted for the `https://` proxy TLS handshake and, because TLS-intercepting proxies re-sign tunneled certificates, folded into the sandbox trust bundle and upstream verification. Requires a proxy URL; the file must exist and hold at least one certificate. |
Through the gateway, the same settings are the `https_proxy`, `no_proxy`,
`proxy_auth_file`, `proxy_auth_allow_insecure`, `proxy_connect_by_hostname`,
and `proxy_ca_bundle` keys under `[openshell.drivers.podman]`; see
`docs/reference/gateway-config.mdx`.
`provider_spiffe_workload_api_socket` accepts either an absolute host UNIX
Workload API socket, projected through a dedicated read-only mount, or an
explicit container-reachable `tcp:IP:port` endpoint. The driver sets the
supervisor's `OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET` accordingly.
`app_armor_profile` shares the canonical
`RuntimeDefault`, `Unconfined`, or `Localhost/<profile>` model with Docker and
Kubernetes. When omitted, the driver sends no override and preserves Podman's
runtime-selected profile. Set `Unconfined` explicitly only when the deployment
requires the supervisor's mount setup to bypass that profile. Explicit confined
choices fail early when Podman reports AppArmor unavailable.
This is an operator-owned egress boundary: the driver passes the settings on
the supervisor's command line, so sandbox and template environment — and any
`ENV` baked into the sandbox image — cannot override them, and the
conventional `HTTPS_PROXY`/`HTTP_PROXY`/`NO_PROXY` variables a sandbox
controls do not steer it. Credentials must be supplied through
`proxy_auth_file`; an inline `user:pass@` in the URL is rejected at startup.
Basic auth over an `http://` proxy is cleartext on the wire: anyone on the
network path between the sandbox host and the proxy can recover the
credential. Setting `proxy_auth_file` therefore requires
`proxy_auth_allow_insecure = true`; both the driver and the in-container
supervisor reject credentials without that explicit acknowledgement.
CONNECT requests target a validated resolved IP by default, so the proxy
performs no DNS resolution and the tunnel stays bound to the address that
passed the sandbox's SSRF and `allowed_ips` checks; the hostname still
travels inside the tunnel (TLS SNI, application `Host`). In split-horizon
networks, point the gateway host at the corporate resolver. Set
`proxy_connect_by_hostname = true` only when the proxy's ACLs filter on
hostnames and reject IP CONNECT targets — it re-opens proxy-side DNS
resolution, making the proxy's ACLs the effective egress control.
## Rootless-Specific Adaptations
The Podman driver is designed for rootless operation. The following adaptations
matter compared to cluster or rootful runtimes:
1. subuid/subgid preflight check: on non-macOS hosts, `check_subuid_range()` in
`driver.rs` warns operators if `/etc/subuid` or `/etc/subgid` entries are
missing for the current user. This is not a hard error because some systems
use LDAP or other mechanisms. macOS skips the check because `podman machine`
runs the Podman service inside a Linux VM.
2. cgroups v2 requirement: the driver refuses to start if cgroups v1 is
detected. Rootless Podman requires the unified cgroup hierarchy.
3. `nsenter` for namespace operations: `openshell-sandbox` uses
`nsenter --net=` instead of `ip netns exec` to avoid the sysfs remount path
that requires real `CAP_SYS_ADMIN` in the host user namespace.
4. `DAC_READ_SEARCH` capability: required for the proxy to read
`/proc/<pid>/fd/` across UIDs within the user namespace.
5. `SETUID` and `SETGID` capabilities: kept from Podman's default capability
set so `drop_privileges()` can call `setuid()` and `setgid()`.
6. `host.containers.internal`: used instead of Docker's `host.docker.internal`
for container-to-host communication. The driver also injects the
OpenShell-owned `host.openshell.internal` alias.
7. Ephemeral port publishing: the SSH compatibility port uses `host_port: 0`
because the bridge network IP is not reliably routable from the host in
rootless mode.
8. tmpfs at `/run/netns`: a private tmpfs lets the supervisor create named
network namespaces via `ip netns add`.
## Implementation References
- Gateway integration: `crates/openshell-gateway/src/lib.rs` registers the
driver factory and constructs `PodmanComputeConfig` from the generic server
build context.
- Server configuration: `crates/openshell-server/src/lib.rs` exposes the
backend-agnostic registry and factory context.
- Gateway relay path: `openshell-core` `Config::sandbox_ssh_socket_path` in
`crates/openshell-core/src/config.rs`.
- SSRF mitigation: `crates/openshell-core/src/net.rs`,
`crates/openshell-sandbox/src/proxy.rs`, and
`crates/openshell-server/src/grpc/policy.rs`.
- Sandbox supervisor: `crates/openshell-sandbox/src/` for Landlock, seccomp,
netns, proxy, and relay behavior shared by all drivers.
- Container engine abstraction: `tasks/scripts/container-engine.sh` for
build/deploy support across Docker and Podman.
- Supervisor image build: `deploy/docker/Dockerfile.supervisor`.
Gateway OTLP configuration continues to export compute-driver spans under the
`openshell-driver-podman` service, preserving gateway trace context.
crates/openshell-driver-podman/src/client.rs
+89
View File
@@ -177,6 +177,8 @@ pub struct ImageInspect {
pub struct ImageConfig {
#[serde(default)]
pub user: String,
#[serde(default)]
pub env: Vec<String>,
}
/// A container summary returned by the list API.
@@ -450,6 +452,93 @@ impl PodmanClient {
.await
}
pub(crate) async fn create_typed_container(
&self,
spec: &(impl serde::Serialize + Sync),
) -> Result<String, PodmanApiError> {
#[derive(serde::Deserialize)]
struct Created {
#[serde(rename = "Id", alias = "ID")]
id: String,
}
let body =
serde_json::to_vec(spec).map_err(|error| PodmanApiError::Json(error.to_string()))?;
let (status, bytes) = self
.request_raw(
hyper::Method::POST,
"/libpod/containers/create",
"application/json",
body.into(),
)
.await?;
if !status.is_success() {
return Err(error_from_response(status.as_u16(), &bytes));
}
let created: Created = serde_json::from_slice(&bytes)
.map_err(|error| PodmanApiError::Json(error.to_string()))?;
validate_name(&created.id)?;
Ok(created.id)
}
pub(crate) async fn copy_to_container(
&self,
name: &str,
destination: &str,
archive: Vec<u8>,
) -> Result<(), PodmanApiError> {
validate_name(name)?;
let (status, bytes) = self
.request_raw(
hyper::Method::PUT,
&format!(
"/libpod/containers/{name}/archive?path={}",
url_encode(destination)
),
"application/x-tar",
archive.into(),
)
.await?;
if status.is_success() {
Ok(())
} else {
Err(error_from_response(status.as_u16(), &bytes))
}
}
pub(crate) async fn verify_isolation_fence(&self, id: &str) -> Result<(), PodmanApiError> {
#[derive(serde::Deserialize)]
#[serde(rename_all = "PascalCase")]
struct HostConfig {
network_mode: String,
privileged: bool,
}
#[derive(serde::Deserialize)]
#[serde(rename_all = "PascalCase")]
struct FenceInspect {
host_config: HostConfig,
network_settings: NetworkSettings,
}
validate_name(id)?;
let inspected: FenceInspect = self
.request_json(
hyper::Method::GET,
&format!("/libpod/containers/{id}/json"),
None,
)
.await?;
if inspected.host_config.network_mode != "none"
|| inspected.host_config.privileged
|| inspected
.network_settings
.networks
.keys()
.any(|name| name != "none")
{
return Err(PodmanApiError::InvalidInput("sandbox requires an unprivileged container with network mode none and no attached networks".into()));
}
Ok(())
}
/// Start a container by name or ID.
pub async fn start_container(&self, name: &str) -> Result<(), PodmanApiError> {
validate_name(name)?;
crates/openshell-driver-podman/src/config.rs
+10 -6
View File
@@ -60,9 +60,11 @@ pub struct PodmanComputeConfig {
pub host_gateway_ip: String,
/// Container stop timeout in seconds (SIGTERM → SIGKILL).
pub stop_timeout_secs: u32,
/// OCI image containing the openshell-sandbox supervisor binary.
/// Mounted read-only into sandbox containers at /opt/openshell/bin
/// using Podman's `type=image` mount.
/// OCI image containing the statically linked `openshell-sandbox` binary.
/// The driver extracts the binary from this image into a verified host
/// cache and mounts it read-only into each sandbox container.
pub sandbox_runtime_image: String,
/// OCI image containing the dynamically linked `openshell-supervisor` binary.
pub supervisor_image: String,
/// Host path to the CA certificate for sandbox mTLS.
///
@@ -91,11 +93,11 @@ pub struct PodmanComputeConfig {
/// Host path to a SPIFFE Workload API Unix socket exposed to sandbox
/// supervisors for provider token exchange client assertions.
pub provider_spiffe_workload_api_socket: Option<PathBuf>,
/// `AppArmor` confinement requested for sandbox containers. Omission sends
/// no override and preserves Podman's runtime-selected profile.
/// `AppArmor` confinement requested for the workload container. Omission
/// sends no override and preserves Podman's runtime-selected profile.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub app_armor_profile: Option<AppArmorProfile>,
/// Health check interval in seconds for sandbox containers.
/// Health check interval in seconds for supervisor containers.
///
/// Podman runs the health check command at this interval to determine
/// container readiness. Lower values detect readiness faster but
@@ -469,6 +471,7 @@ impl Default for PodmanComputeConfig {
network_name: DEFAULT_NETWORK_NAME.to_string(),
host_gateway_ip: Self::default_host_gateway_ip(),
stop_timeout_secs: DEFAULT_PODMAN_STOP_TIMEOUT_SECS,
sandbox_runtime_image: openshell_core::config::default_sandbox_runtime_image(),
supervisor_image: openshell_core::config::default_supervisor_image(),
guest_tls_ca: None,
guest_tls_cert: None,
@@ -503,6 +506,7 @@ impl std::fmt::Debug for PodmanComputeConfig {
.field("network_name", &self.network_name)
.field("host_gateway_ip", &self.host_gateway_ip)
.field("stop_timeout_secs", &self.stop_timeout_secs)
.field("sandbox_runtime_image", &self.sandbox_runtime_image)
.field("supervisor_image", &self.supervisor_image)
.field("guest_tls_ca", &self.guest_tls_ca)
.field("guest_tls_cert", &self.guest_tls_cert)
crates/openshell-driver-podman/src/container.rs
+459 -237
View File
@@ -197,7 +197,7 @@ pub fn short_id(id: &str) -> String {
// ---------------------------------------------------------------------------
#[derive(Serialize)]
struct ContainerSpec {
pub struct ContainerSpec {
name: String,
image: String,
labels: BTreeMap<String, String>,
@@ -212,18 +212,21 @@ struct ContainerSpec {
entrypoint: Vec<String>,
command: Vec<String>,
user: String,
#[serde(skip_serializing_if = "Vec::is_empty")]
groups: Vec<String>,
#[serde(skip_serializing_if = "Vec::is_empty")]
unsetenv: Vec<String>,
cap_drop: Vec<String>,
cap_add: Vec<String>,
no_new_privileges: bool,
seccomp_profile_path: String,
/// Podman's container create API accepts `AppArmor` through the dedicated
/// `apparmor_profile` `SpecGenerator` field. This is not Docker's
/// `security_opt` representation.
#[serde(skip_serializing_if = "Option::is_none")]
apparmor_profile: Option<String>,
#[serde(skip_serializing_if = "String::is_empty")]
seccomp_profile_path: String,
#[serde(skip_serializing_if = "BTreeMap::is_empty")]
sysctl: BTreeMap<String, String>,
image_pull_policy: String,
#[serde(skip_serializing_if = "Option::is_none")]
healthconfig: Option<HealthConfig>,
healthconfig: HealthConfig,
resource_limits: ResourceLimits,
/// Env-type secrets: map of `ENV_VAR_NAME → secret_name`.
/// Podman's libpod `SpecGenerator` uses `secret_env` (a flat map) for
@@ -340,8 +343,6 @@ struct SecretMount {
struct ResourceLimits {
cpu: CpuLimits,
memory: MemoryLimits,
// Podman's libpod API consumes the OCI LinuxResources shape. A Docker-style
// scalar PidsLimit is silently ignored and leaves the runtime default.
#[serde(skip_serializing_if = "Option::is_none")]
pids: Option<PidsLimits>,
}
@@ -486,7 +487,7 @@ fn build_env(
config: &PodmanComputeConfig,
image: &str,
oci_user: &str,
) -> BTreeMap<String, String> {
) -> Result<BTreeMap<String, String>, ComputeDriverError> {
let spec = sandbox.spec.as_ref();
let template = spec.and_then(|s| s.template.as_ref());
@@ -511,10 +512,11 @@ fn build_env(
user_env.insert(k.clone(), v.clone());
}
}
env.extend(user_env.clone());
if !user_env.is_empty()
&& let Ok(json) = serde_json::to_string(&user_env)
{
// User environment belongs exclusively to mediated workload children. In
// particular, never activate loader or policy overrides in the supervisor.
if !user_env.is_empty() {
let json = serde_json::to_string(&user_env)
.map_err(|error| ComputeDriverError::Precondition(error.to_string()))?;
env.insert(openshell_core::sandbox_env::USER_ENVIRONMENT.into(), json);
}
@@ -543,7 +545,7 @@ fn build_env(
);
env.insert("OPENSHELL_CONTAINER_IMAGE".into(), image.to_string());
let main_process = openshell_core::sandbox_env::MainProcessConfig::encode_driver_spec(spec)
.expect("main process config serialization cannot fail");
.map_err(|error| ComputeDriverError::Precondition(error.to_string()))?;
env.insert(
openshell_core::sandbox_env::MAIN_PROCESS_SPEC.into(),
main_process,
@@ -615,7 +617,7 @@ fn build_env(
);
}
env
Ok(env)
}
/// Merge labels from the sandbox template with required managed labels.
@@ -665,12 +667,14 @@ fn build_resource_limits(sandbox: &DriverSandbox, config: &PodmanComputeConfig)
period: DEFAULT_CPU_PERIOD,
},
memory: MemoryLimits { limit: mem_bytes },
pids: config
.sandbox_pids_limit
.map(|limit| PidsLimits { limit: limit.get() }),
pids: podman_pids_limit(config.sandbox_pids_limit).map(|limit| PidsLimits { limit }),
}
}
fn podman_pids_limit(value: Option<std::num::NonZeroI64>) -> Option<i64> {
value.map(std::num::NonZeroI64::get)
}
pub fn podman_driver_volume_mount_sources(
sandbox: &DriverSandbox,
enable_bind_mounts: bool,
@@ -880,6 +884,7 @@ fn validate_podman_driver_mounts(
}
};
driver_mounts::validate_container_mount_target(target)?;
driver_mounts::validate_mount_control_path(target, "/.openshell")?;
let normalized_target = driver_mounts::normalize_mount_target(target);
if !targets.insert(normalized_target.clone()) {
return Err(format!(
@@ -951,14 +956,6 @@ fn validate_tmpfs_options(options: &[String]) -> Result<Vec<String>, String> {
.collect()
}
fn podman_apparmor_profile(profile: Option<&openshell_core::AppArmorProfile>) -> Option<String> {
match profile {
None | Some(openshell_core::AppArmorProfile::RuntimeDefault) => None,
Some(openshell_core::AppArmorProfile::Unconfined) => Some("unconfined".to_string()),
Some(openshell_core::AppArmorProfile::Localhost(profile)) => Some(profile.clone()),
}
}
/// Build the Podman container creation JSON spec.
#[cfg(test)]
#[must_use]
@@ -1025,6 +1022,7 @@ pub fn build_container_spec_with_token_and_gpu_devices(
}
#[allow(clippy::too_many_arguments)]
#[cfg(test)]
pub fn build_container_spec_for_image(
sandbox: &DriverSandbox,
config: &PodmanComputeConfig,
@@ -1036,10 +1034,36 @@ pub fn build_container_spec_for_image(
supervisor_bin_path: Option<&Path>,
tls_secret_names: Option<&[String; 3]>,
) -> Result<Value, ComputeDriverError> {
serde_json::to_value(build_base_spec(
sandbox,
config,
token_secret_name,
gpu_device_ids,
requested_image,
image_id,
oci_user,
supervisor_bin_path,
tls_secret_names,
)?)
.map_err(|error| ComputeDriverError::Message(format!("encode Podman spec: {error}")))
}
#[allow(clippy::too_many_arguments)]
fn build_base_spec(
sandbox: &DriverSandbox,
config: &PodmanComputeConfig,
token_secret_name: Option<&str>,
gpu_device_ids: Option<&[String]>,
requested_image: &str,
image_id: &str,
oci_user: &str,
supervisor_bin_path: Option<&Path>,
tls_secret_names: Option<&[String; 3]>,
) -> Result<ContainerSpec, ComputeDriverError> {
let name = container_name(&sandbox.workspace, &sandbox.name, &sandbox.id);
let vol = volume_name(&sandbox.id);
let env = build_env(sandbox, config, requested_image, oci_user);
let env = build_env(sandbox, config, requested_image, oci_user)?;
let labels = build_labels(sandbox);
let resource_limits = build_resource_limits(sandbox, config);
let user_mounts = podman_user_mounts(sandbox, config.enable_bind_mounts)
@@ -1080,7 +1104,7 @@ pub fn build_container_spec_for_image(
Vec::new()
} else {
vec![ImageVolume {
source: config.supervisor_image.clone(),
source: config.sandbox_runtime_image.clone(),
destination: SUPERVISOR_MOUNT_DIR.into(),
rw: false,
}]
@@ -1098,10 +1122,10 @@ pub fn build_container_spec_for_image(
labels,
env,
volumes,
// Side-load the supervisor binary from a standalone OCI image.
// Side-load the sandbox runtime binary from its standalone OCI image.
// Podman resolves image_volumes at the libpod layer, mounting the
// image's filesystem at the destination path without starting a
// container from it. The supervisor image exposes the binary at
// container from it. The sandbox runtime image exposes the binary at
// /openshell-sandbox, so it appears at /opt/openshell/bin/openshell-sandbox.
image_volumes,
hostname: format!("sandbox-{}", sandbox.name),
@@ -1117,86 +1141,19 @@ pub fn build_container_spec_for_image(
// corporate proxy flags follow it; the workload command comes from
// the reserved environment variable.
command,
// Force the supervisor to run as root (UID 0). Sandbox images may
// set a non-root USER directive (e.g. `USER sandbox`), but the
// supervisor needs root to create network namespaces, set up the
// proxy, and configure Landlock/seccomp. This matches the K8s
// driver's runAsUser: 0.
user: "0:0".into(),
// Podman's default container capability set is already restricted:
// CHOWN DAC_OVERRIDE FOWNER FSETID KILL SETGID SETUID SETPCAP
// NET_BIND_SERVICE SYS_CHROOT SETFCAP
// We add what the supervisor needs and drop what it doesn't.
cap_drop: vec![
// Not needed: standard file permission bits are sufficient; dropping
// prevents the supervisor from bypassing DAC checks it shouldn't need.
"DAC_OVERRIDE".into(),
// Not needed: the supervisor does not create setuid/setgid executables.
"FSETID".into(),
// Not needed: the supervisor does not bind privileged ports (<1024).
"NET_BIND_SERVICE".into(),
// Not in Podman's default set but explicitly denied in case the image
// or runtime adds it; raw sockets are not required.
"NET_RAW".into(),
// Not needed: the supervisor does not manipulate file capabilities.
"SETFCAP".into(),
// Not needed: the supervisor does not call chroot().
"SYS_CHROOT".into(),
],
cap_add: vec![
// seccomp filter installation, namespace creation, Landlock setup.
"SYS_ADMIN".into(),
// Network namespace veth setup, IP/route configuration.
"NET_ADMIN".into(),
// Reading /proc/<pid>/exe and ancestor walk for process identity in policy.
"SYS_PTRACE".into(),
// Reading /dev/kmsg for bypass-detection diagnostics.
"SYSLOG".into(),
// Reading /proc/<pid>/fd/ across UIDs for process identity resolution.
// In rootless Podman the supervisor runs as UID 0 inside a user namespace
// while sandbox processes run as the sandbox user. The kernel's
// proc_fd_permission() calls generic_permission() which denies cross-UID
// access to the dr-x------ fd directory unless this cap is present.
// Without it the proxy cannot determine which binary made each outbound
// connection and all traffic is denied.
"DAC_READ_SEARCH".into(),
// Child setup clears the capability bounding set before exec, which
// requires CAP_SETPCAP in the supervisor until drop_privileges().
"SETPCAP".into(),
// Forwarding shutdown signals to the canonical workload process
// group after it drops to the sandbox UID requires CAP_KILL.
"KILL".into(),
],
// SETUID, SETGID, SETPCAP, CHOWN, and FOWNER are intentionally kept from
// Podman's default set and not dropped:
// SETUID/SETGID – drop_privileges(): setuid()/setgid()/initgroups() to the
// sandbox user. In rootless Podman cap_drop:ALL removes them
// from the bounding set even though uid=0 owns the user
// namespace — so we keep them by not dropping them explicitly.
// SETPCAP – drop_privileges(): clears the child capability
// bounding set before the sandbox user execs.
// CHOWN – prepare_filesystem(): chown(path, uid, gid) on newly
// created read_write directories so the sandbox user can
// write to them.
// FOWNER – chown on files where the supervisor is not the owner
// (e.g. pre-existing directories owned by another user).
//
// Disable the container-level seccomp profile. The sandbox supervisor The sandbox supervisor
// installs its own policy-aware BPF seccomp filter at runtime via
// seccompiler (two-phase: clone3 blocker + main filter). The runtime
// filter is more restrictive than Podman's default — it blocks 20+
// dangerous syscalls and conditionally restricts socket domains based
// on network policy. The filter self-seals by blocking further
// seccomp(SET_MODE_FILTER) calls after installation.
//
// A container-level profile would interfere by blocking the landlock
// and seccomp syscalls the supervisor needs during setup, before it
// locks itself down.
// The paired builder supplies the immutable non-root identity.
user: String::new(),
groups: Vec::new(),
unsetenv: Vec::new(),
cap_drop: vec!["ALL".into()],
cap_add: Vec::new(),
no_new_privileges: true,
seccomp_profile_path: "unconfined".into(),
apparmor_profile: podman_apparmor_profile(config.app_armor_profile.as_ref()),
apparmor_profile: None,
// Omission selects the runtime default, never an unconfined profile.
seccomp_profile_path: String::new(),
sysctl: BTreeMap::new(),
image_pull_policy: "never".to_string(),
healthconfig: config.health_check_interval_secs.map(|interval_secs| HealthConfig {
healthconfig: HealthConfig {
test: vec![
"CMD-SHELL".into(),
format!(
@@ -1205,11 +1162,14 @@ pub fn build_container_spec_for_image(
openshell_core::config::DEFAULT_SSH_PORT
),
],
interval: interval_secs.get() * 1_000_000_000,
interval: config
.health_check_interval_secs
.map_or(10, std::num::NonZeroU64::get)
* 1_000_000_000,
timeout: 2_000_000_000,
retries: 10,
start_period: 5_000_000_000,
}),
},
resource_limits,
secret_env: BTreeMap::new(),
secrets: {
@@ -1409,7 +1369,211 @@ pub fn build_container_spec_for_image(
},
};
Ok(serde_json::to_value(container_spec).expect("ContainerSpec serialization cannot fail"))
Ok(container_spec)
}
/// Driver-owned inputs for the two independent runtime containers.
pub struct IsolationSpecInput<'a> {
pub sandbox: &'a DriverSandbox,
pub config: &'a PodmanComputeConfig,
pub token_secret: Option<&'a str>,
pub gpu_devices: Option<&'a [String]>,
pub requested_image: &'a str,
pub image_id: &'a str,
pub image_user: &'a str,
pub image_env: &'a [String],
pub supervisor_bin: Option<&'a Path>,
pub tls_secrets: Option<&'a [String; 3]>,
pub identity: &'a openshell_isolation_interface::contract::ResolvedWorkloadIdentity,
}
pub struct IsolationSpecs {
pub workload: ContainerSpec,
pub supervisor: ContainerSpec,
}
pub fn build_isolation_specs(
input: IsolationSpecInput<'_>,
) -> Result<IsolationSpecs, ComputeDriverError> {
let base = || {
build_base_spec(
input.sandbox,
input.config,
input.token_secret,
input.gpu_devices,
input.requested_image,
input.image_id,
input.image_user,
input.supervisor_bin,
input.tls_secrets,
)
};
let mut workload = base()?;
let mut supervisor = base()?;
let user = format!("{}:{}", input.identity.uid, input.identity.gid);
let channel = crate::isolation::channel_volume_name(&input.sandbox.id);
workload
.labels
.insert(crate::isolation::LABEL_ROLE.into(), "sandbox".into());
workload.env = BTreeMap::new();
workload.unsetenv = input
.image_env
.iter()
.filter_map(|entry| entry.split_once('=').map(|(key, _)| key.to_string()))
.collect();
workload.command = vec![
"--bootstrap".into(),
crate::isolation::BOOTSTRAP_PATH.into(),
];
workload.user.clone_from(&user);
workload.groups = input
.identity
.supplementary_gids
.iter()
.map(ToString::to_string)
.collect();
workload.cap_drop = vec!["ALL".into()];
workload.cap_add.clear();
workload.apparmor_profile = input
.config
.app_armor_profile
.as_ref()
.and_then(openshell_core::config::AppArmorProfile::oci_security_opt)
.and_then(|option| option.strip_prefix("apparmor=").map(str::to_string));
workload.seccomp_profile_path.clear();
workload
.sysctl
.insert("net.ipv4.ip_unprivileged_port_start".into(), "0".into());
workload.netns.nsmode = "none".into();
workload.networks.clear();
workload.portmappings.clear();
workload.hostadd.clear();
workload.secret_env.clear();
workload.secrets.clear();
workload.healthconfig.test = vec!["NONE".into()];
workload
.mounts
.retain(|mount| !trusted_mount(&mount.destination));
workload.mounts.push(Mount {
kind: "tmpfs".into(),
source: "tmpfs".into(),
destination: openshell_sandbox_backend::SUPERVISOR_CA_RUNTIME_ROOT.into(),
options: vec![
"rw".into(),
"noexec".into(),
"nosuid".into(),
"nodev".into(),
// Libpod's OCI `mounts` API rejects tmpfs uid/gid options. The
// unprivileged runtime creates the owned material subdirectory.
"mode=0777".into(),
"size=1m".into(),
],
});
workload.volumes.push(NamedVolume {
name: channel.clone(),
dest: crate::isolation::CHANNEL_ROOT.into(),
options: vec!["rw".into(), "z".into()],
});
supervisor.name = crate::isolation::supervisor_name(&input.sandbox.id);
supervisor
.labels
.insert(crate::isolation::LABEL_ROLE.into(), "supervisor".into());
supervisor.image.clone_from(&input.config.supervisor_image);
supervisor.entrypoint = vec!["/openshell-supervisor".into()];
supervisor.command.extend([
format!(
"--backend-descriptor-file={}",
crate::isolation::RUNTIME_DESCRIPTOR_PATH
),
format!("--auth-bundle-file={}", crate::isolation::AUTH_BUNDLE_PATH),
"--health-socket-path=/run/openshell/supervisor-health.sock".into(),
]);
supervisor.env.insert(
openshell_core::sandbox_env::ADMITTED_ISOLATION_BACKEND.into(),
openshell_sandbox_backend::BACKEND_NAME.into(),
);
supervisor.user = user;
supervisor.groups = input
.identity
.supplementary_gids
.iter()
.map(ToString::to_string)
.collect();
supervisor.cap_drop = vec!["ALL".into()];
supervisor.cap_add.clear();
supervisor.seccomp_profile_path.clear();
// The trusted supervisor originates approved egress from the Podman host
// network. Keep it in the caller's user namespace as well: joining the
// workload's user namespace is incompatible with host networking under
// rootless Podman, and the authenticated channel does not require a shared
// user namespace. The workload remains fenced by network=none.
supervisor.userns = Some(UserNS {
nsmode: "host".into(),
value: None,
});
supervisor.idmappings = None;
supervisor.netns.nsmode = "host".into();
supervisor.networks.clear();
supervisor.portmappings.clear();
supervisor.devices = None;
supervisor.image_volumes.clear();
supervisor.volumes = vec![NamedVolume {
name: channel,
dest: crate::isolation::CHANNEL_ROOT.into(),
options: vec!["ro".into(), "z".into()],
}];
supervisor.mounts.retain(|mount| {
trusted_mount(&mount.destination)
&& mount.destination != openshell_core::container_paths::NETNS_MOUNT_ROOT
});
for destination in ["/run", "/var/log", "/tmp"] {
supervisor.mounts.push(Mount {
kind: "tmpfs".into(),
source: "tmpfs".into(),
destination: destination.into(),
options: vec![
"rw".into(),
"noexec".into(),
"nosuid".into(),
"nodev".into(),
"mode=0777".into(),
"size=64m".into(),
],
});
}
for secret in &mut supervisor.secrets {
secret.uid = input.identity.uid;
secret.gid = input.identity.gid;
}
if let Some(interval) = input.config.health_check_interval_secs {
supervisor.healthconfig.test = vec![
"CMD".into(),
"/openshell-supervisor".into(),
"health".into(),
"--socket".into(),
"/run/openshell/supervisor-health.sock".into(),
];
supervisor.healthconfig.interval = interval.get() * 1_000_000_000;
} else {
supervisor.healthconfig.test = vec!["NONE".into()];
}
Ok(IsolationSpecs {
workload,
supervisor,
})
}
fn trusted_mount(destination: &str) -> bool {
matches!(
destination,
TLS_CA_MOUNT_PATH
| TLS_CERT_MOUNT_PATH
| TLS_KEY_MOUNT_PATH
| PROXY_CA_MOUNT_PATH
| PROVIDER_SPIFFE_WORKLOAD_API_SOCKET_MOUNT_DIR
) || destination == openshell_core::container_paths::NETNS_MOUNT_ROOT
}
fn provider_spiffe_workload_api_socket_env_value(config: &PodmanComputeConfig) -> Option<String> {
@@ -1516,6 +1680,129 @@ mod tests {
static ENV_LOCK: std::sync::LazyLock<std::sync::Mutex<()>> =
std::sync::LazyLock::new(|| std::sync::Mutex::new(()));
#[test]
fn isolated_pair_keeps_privileges_network_and_secrets_out_of_workload() {
let sandbox = DriverSandbox {
id: "pair".into(),
name: "agent".into(),
..Default::default()
};
let mut config = PodmanComputeConfig::default();
config.app_armor_profile = Some(openshell_core::config::AppArmorProfile::Localhost(
"openshell-sandbox".into(),
));
config.userns = Some("auto".into());
let identity = openshell_isolation_interface::contract::ResolvedWorkloadIdentity::new(
1000,
1001,
vec![2000],
"image".into(),
"sha256:image".into(),
)
.unwrap();
let env = vec![
"LD_PRELOAD=/hostile.so".into(),
"HTTP_PROXY=http://bypass".into(),
];
let specs = build_isolation_specs(IsolationSpecInput {
sandbox: &sandbox,
config: &config,
token_secret: Some("jwt"),
gpu_devices: None,
requested_image: "image:latest",
image_id: "sha256:image",
image_user: "1000:1001",
image_env: &env,
supervisor_bin: None,
tls_secrets: None,
identity: &identity,
})
.unwrap();
for spec in [&specs.workload, &specs.supervisor] {
assert_eq!(spec.user, "1000:1001");
assert_eq!(spec.groups, vec!["2000"]);
assert_eq!(spec.cap_drop, vec!["ALL"]);
assert!(spec.cap_add.is_empty());
assert!(spec.seccomp_profile_path.is_empty());
assert!(spec.no_new_privileges);
}
assert_eq!(specs.workload.netns.nsmode, "none");
assert_eq!(
specs
.workload
.userns
.as_ref()
.map(|userns| userns.nsmode.as_str()),
Some("auto")
);
assert_eq!(
specs.workload.apparmor_profile.as_deref(),
Some("openshell-sandbox")
);
assert_eq!(specs.supervisor.apparmor_profile, None);
let workload_json = serde_json::to_string(&specs.workload).unwrap();
assert!(workload_json.contains("\"apparmor_profile\":\"openshell-sandbox\""));
assert_eq!(specs.supervisor.healthconfig.test, vec!["NONE"]);
assert!(specs.workload.networks.is_empty());
assert!(specs.workload.portmappings.is_empty());
assert_eq!(specs.supervisor.netns.nsmode, "host");
assert_eq!(
specs
.supervisor
.userns
.as_ref()
.map(|userns| userns.nsmode.as_str()),
Some("host")
);
assert!(specs.supervisor.idmappings.is_none());
assert!(specs.supervisor.networks.is_empty());
assert!(specs.supervisor.portmappings.is_empty());
assert!(specs.workload.env.is_empty());
assert_eq!(specs.workload.unsetenv, vec!["LD_PRELOAD", "HTTP_PROXY"]);
assert!(specs.workload.secrets.is_empty());
assert!(
specs
.workload
.mounts
.iter()
.all(|mount| !trusted_mount(&mount.destination))
);
let supervisor_ca_mount = specs
.workload
.mounts
.iter()
.find(|mount| {
mount.destination == openshell_sandbox_backend::SUPERVISOR_CA_RUNTIME_ROOT
})
.expect("workload supervisor CA mount");
assert_eq!(supervisor_ca_mount.kind, "tmpfs");
assert_eq!(supervisor_ca_mount.source, "tmpfs");
for option in ["rw", "noexec", "nosuid", "nodev", "mode=0777", "size=1m"] {
assert!(supervisor_ca_mount.options.contains(&option.to_string()));
}
assert!(
specs
.workload
.mounts
.iter()
.all(|mount| mount.destination != "/run")
);
assert_eq!(specs.supervisor.secrets.len(), 1);
assert_eq!(specs.supervisor.secrets[0].source, "jwt");
assert_eq!(specs.supervisor.secrets[0].uid, 1000);
assert_eq!(specs.supervisor.volumes.len(), 1);
assert!(specs.workload.volumes[0].options.contains(&"rw".into()));
assert!(!specs.workload.volumes[0].options.contains(&"nocopy".into()));
assert!(specs.supervisor.volumes[0].options.contains(&"ro".into()));
assert!(specs.supervisor.volumes[0].options.contains(&"z".into()));
assert!(
!specs.supervisor.volumes[0]
.options
.contains(&"nocopy".into())
);
assert_eq!(specs.supervisor.entrypoint, vec!["/openshell-supervisor"]);
}
fn json_struct(value: Value) -> prost_types::Struct {
let Value::Object(object) = value else {
panic!("expected JSON object");
@@ -1564,7 +1851,7 @@ mod tests {
}
#[test]
fn container_spec_applies_resource_limits() {
fn container_spec_applies_cpu_and_memory_limits() {
use openshell_core::proto::compute::v1::{
DriverResourceRequirements, DriverSandboxSpec, DriverSandboxTemplate,
};
@@ -1581,22 +1868,18 @@ mod tests {
}),
..Default::default()
});
let mut config = test_config();
config.sandbox_pids_limit = std::num::NonZeroI64::new(2048);
let spec = build_container_spec(&sandbox, &config);
let config = test_config();
let limits = build_resource_limits(&sandbox, &config);
assert_eq!(limits.cpu.quota, 50_000);
assert_eq!(limits.memory.limit, 2 * 1024 * 1024 * 1024);
assert_eq!(
spec["resource_limits"]["cpu"]["quota"].as_u64(),
Some(50_000)
);
assert_eq!(
spec["resource_limits"]["memory"]["limit"].as_u64(),
Some(2 * 1024 * 1024 * 1024)
);
assert_eq!(
spec["resource_limits"]["pids"]["limit"].as_i64(),
Some(2048)
limits.pids.as_ref().map(|pids| pids.limit),
openshell_core::config::default_sandbox_pids_limit().map(std::num::NonZeroI64::get)
);
let serialized = serde_json::to_string(&limits).unwrap();
assert!(serialized.contains("\"pids\":{\"limit\":2048}"));
assert!(!serialized.contains("PidsLimit"));
}
#[test]
@@ -1604,43 +1887,9 @@ mod tests {
let sandbox = test_sandbox("test-id", "test-name");
let mut config = test_config();
config.sandbox_pids_limit = None;
let spec = build_container_spec(&sandbox, &config);
let limits = build_resource_limits(&sandbox, &config);
assert!(spec["resource_limits"].get("pids").is_none());
}
#[test]
fn container_spec_uses_podman_apparmor_profile_field() {
let sandbox = test_sandbox("test-id", "test-name");
for (profile, expected) in [
(openshell_core::AppArmorProfile::Unconfined, "unconfined"),
(
openshell_core::AppArmorProfile::Localhost("openshell-supervisor".to_string()),
"openshell-supervisor",
),
] {
let mut config = test_config();
config.app_armor_profile = Some(profile);
let spec = build_container_spec(&sandbox, &config);
assert_eq!(spec["apparmor_profile"].as_str(), Some(expected));
assert!(spec.get("security_opt").is_none());
}
}
#[test]
fn container_spec_omits_podman_apparmor_profile_for_runtime_default() {
let sandbox = test_sandbox("test-id", "test-name");
for profile in [None, Some(openshell_core::AppArmorProfile::RuntimeDefault)] {
let mut config = test_config();
config.app_armor_profile = profile;
let spec = build_container_spec(&sandbox, &config);
assert!(spec.get("apparmor_profile").is_none());
assert!(spec.get("security_opt").is_none());
}
assert!(limits.pids.is_none());
}
#[test]
@@ -1681,7 +1930,8 @@ mod tests {
container["env"]["OPENSHELL_CONTAINER_IMAGE"].as_str(),
Some("registry.example/app:latest")
);
assert_eq!(container["user"].as_str(), Some("0:0"));
// Only the paired builder materializes the immutable non-root user.
assert_eq!(container["user"].as_str(), Some(""));
assert_eq!(container["image_pull_policy"].as_str(), Some("never"));
assert_eq!(container["dns_search"], serde_json::json!([]));
assert_eq!(container["dns_option"], serde_json::json!([]));
@@ -1921,66 +2171,25 @@ mod tests {
}
#[test]
fn container_spec_includes_required_capabilities() {
fn container_spec_defaults_drop_capabilities_and_keep_runtime_seccomp() {
let sandbox = test_sandbox("test-id", "test-name");
let config = test_config();
let spec = build_container_spec(&sandbox, &config);
let added: Vec<&str> = spec["cap_add"]
.as_array()
.expect("cap_add should be an array")
.iter()
.filter_map(|v| v.as_str())
.collect();
assert!(added.contains(&"SYS_ADMIN"), "missing SYS_ADMIN");
assert!(added.contains(&"NET_ADMIN"), "missing NET_ADMIN");
assert!(added.contains(&"SYS_PTRACE"), "missing SYS_PTRACE");
assert!(added.contains(&"SYSLOG"), "missing SYSLOG");
assert!(
added.contains(&"DAC_READ_SEARCH"),
"missing DAC_READ_SEARCH"
);
assert!(added.contains(&"SETPCAP"), "missing SETPCAP");
assert!(added.contains(&"KILL"), "missing KILL");
// SETUID and SETGID are NOT in cap_add — they remain available from the
// default bounding set because we no longer use cap_drop:ALL. Verify they
// are also not explicitly dropped. Similarly SETPCAP, CHOWN and FOWNER
// must not be dropped because child setup clears the bounding set and
// prepare_filesystem() calls chown() on newly created read_write
// directories before the supervisor drops privileges.
let dropped: Vec<&str> = spec["cap_drop"]
.as_array()
.expect("cap_drop should be an array")
.iter()
.filter_map(|v| v.as_str())
.collect();
assert!(!dropped.contains(&"SETUID"), "SETUID must not be dropped");
assert!(!dropped.contains(&"SETGID"), "SETGID must not be dropped");
assert!(
dropped.contains(&"NET_BIND_SERVICE"),
"NET_BIND_SERVICE must stay dropped; policy DNS binds an unprivileged port"
);
assert!(
!dropped.contains(&"CHOWN"),
"CHOWN must not be dropped (needed for prepare_filesystem chown)"
);
assert!(
!dropped.contains(&"FOWNER"),
"FOWNER must not be dropped (needed for chown on non-owned files)"
);
assert!(
!dropped.contains(&"SETPCAP"),
"SETPCAP must not be dropped (needed for child bounding-set clear)"
);
assert!(
!dropped.contains(&"KILL"),
"KILL must not be dropped (needed to signal the sandbox workload on shutdown)"
);
assert!(
!dropped.contains(&"ALL"),
"must not use cap_drop:ALL in rootless Podman"
);
let spec = build_base_spec(
&sandbox,
&config,
None,
None,
"image",
"sha256:image",
"",
None,
None,
)
.unwrap();
assert_eq!(spec.cap_drop, vec!["ALL"]);
assert!(spec.cap_add.is_empty());
assert!(spec.seccomp_profile_path.is_empty());
assert!(spec.no_new_privileges);
}
#[test]
@@ -2016,8 +2225,7 @@ mod tests {
#[test]
fn container_spec_healthcheck_accepts_supervisor_socket() {
let sandbox = test_sandbox("test-id", "test-name");
let mut config = test_config();
config.health_check_interval_secs = std::num::NonZeroU64::new(10);
let config = test_config();
let spec = build_container_spec(&sandbox, &config);
let healthcheck = spec["healthconfig"]["test"]
@@ -2033,13 +2241,6 @@ mod tests {
);
}
#[test]
fn container_spec_omits_healthcheck_when_disabled() {
let sandbox = test_sandbox("test-id", "test-name");
let spec = build_container_spec(&sandbox, &test_config());
assert!(spec.get("healthconfig").is_none());
}
#[test]
fn container_spec_healthcheck_interval_from_config() {
let sandbox = test_sandbox("test-id", "test-name");
@@ -2476,7 +2677,7 @@ mod tests {
}
#[test]
fn container_spec_includes_supervisor_image_volume() {
fn container_spec_includes_sandbox_runtime_image_volume() {
let sandbox = test_sandbox("test-id", "test-name");
let config = test_config();
let spec = build_container_spec(&sandbox, &config);
@@ -2493,8 +2694,8 @@ mod tests {
let vol = &image_volumes[0];
assert_eq!(
vol["source"].as_str(),
Some(openshell_core::config::default_supervisor_image().as_str()),
"image volume source should be the supervisor image"
Some(openshell_core::config::default_sandbox_runtime_image().as_str()),
"image volume source should be the sandbox runtime image"
);
assert_eq!(
vol["destination"].as_str(),
@@ -2579,9 +2780,9 @@ mod tests {
let image_volumes = spec["image_volumes"]
.as_array()
.expect("image_volumes should be an array");
let expected_supervisor = openshell_core::config::default_supervisor_image();
let expected_sandbox_runtime = openshell_core::config::default_sandbox_runtime_image();
assert!(image_volumes.iter().any(|volume| {
volume["source"].as_str() == Some(expected_supervisor.as_str())
volume["source"].as_str() == Some(expected_sandbox_runtime.as_str())
&& volume["destination"].as_str() == Some("/opt/openshell/bin")
}));
assert!(image_volumes.iter().any(|volume| {
@@ -2930,6 +3131,27 @@ mod tests {
assert!(err.to_string().contains("reserved OpenShell path"));
}
#[test]
fn user_mounts_cannot_replace_private_channel_hierarchy() {
for target in [
"/.openshell",
"/.openshell/channel",
"/.openshell/channel/sandbox",
"/.openshell/supervisor",
] {
let mount = PodmanDriverMountConfig::Volume {
source: "user-owned".into(),
target: target.into(),
read_only: false,
subpath: None,
};
assert!(
validate_podman_driver_mounts(&[mount], false).is_err(),
"{target}"
);
}
}
#[test]
fn container_spec_uses_configured_host_gateway_ip() {
let sandbox = test_sandbox("test-id", "test-name");
@@ -3367,7 +3589,7 @@ mod tests {
!image_volumes
.iter()
.any(|v| v["destination"].as_str() == Some(SUPERVISOR_MOUNT_DIR)),
"supervisor image volume should not be present when bind path is provided"
"sandbox runtime image volume should not be present when bind path is provided"
);
let mounts = spec["mounts"]
@@ -3401,7 +3623,7 @@ mod tests {
image_volumes
.iter()
.any(|v| v["destination"].as_str() == Some(SUPERVISOR_MOUNT_DIR)),
"supervisor image volume should be present by default"
"sandbox runtime image volume should be present by default"
);
let mounts = spec["mounts"]
crates/openshell-driver-podman/src/driver.rs
+851 -203
View File
File diff suppressed because it is too large Load Diff
crates/openshell-driver-podman/src/grpc.rs
+10 -5
View File
@@ -162,8 +162,7 @@ impl ComputeDriver for ComputeDriverService {
.into_inner()
.sandbox
.ok_or_else(|| Status::invalid_argument("sandbox is required"))?;
self.driver
.create_sandbox(&sandbox)
Box::pin(self.driver.create_sandbox(&sandbox))
.await
.map_err(Status::from)?;
Ok(Response::new(CreateSandboxResponse {}))
@@ -201,7 +200,11 @@ impl ComputeDriver for ComputeDriverService {
return Err(Status::invalid_argument("sandbox_id is required"));
}
self.driver
.start_sandbox(&request.sandbox_id)
.start_sandbox(
&request.sandbox_id,
&request.generation_id,
&request.launch_authentication,
)
.await
.map_err(Status::from)?;
Ok(Response::new(StartSandboxResponse {}))
@@ -680,6 +683,8 @@ mod tests {
let (socket_path, request_log, handle) = spawn_podman_stub(
"forward-id",
vec![
StubResponse::new(StatusCode::NO_CONTENT, ""), // companion
StubResponse::new(StatusCode::NO_CONTENT, ""), // channel
// list_containers returns empty (container already gone)
StubResponse::new(StatusCode::OK, "[]"),
// remove_volume
@@ -708,9 +713,9 @@ mod tests {
.lock()
.expect("request log lock should not be poisoned")
.clone();
assert!(requests[0].contains("/libpod/containers/json"));
assert!(requests[2].contains("/libpod/containers/json"));
assert_eq!(
requests[1],
requests[3],
format!(
"DELETE {}",
api_path(&format!("/libpod/volumes/{volume_name}"))
crates/openshell-driver-podman/src/isolation.rs
+463
View File
@@ -0,0 +1,463 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
//! Podman-owned provisioning for the common authenticated isolation channel.
use std::collections::{BTreeMap, HashMap};
#[cfg(test)]
use std::io::Read;
use std::path::PathBuf;
use openshell_core::ComputeDriverError;
use openshell_core::proto::compute::v1::DriverSandbox;
use openshell_isolation_interface::contract::{DriverFenceEvidence, ResolvedWorkloadIdentity};
use openshell_sandbox_backend::boundary_protocol::{
BoundaryConfig, BoundaryListener, GatewayVerificationKey, SandboxRuntimeDescriptor,
SandboxTlsClientConfig, SandboxTlsServerConfig, SandboxTransport,
generate_sandbox_tls_material,
};
use serde::{Deserialize, Serialize};
pub const LABEL_ROLE: &str = "openshell.io/isolation-role";
pub const WORKLOAD_FILTER: &str = "openshell.io/isolation-role=sandbox";
pub const CHANNEL_ROOT: &str = "/.openshell/channel";
pub const BOOTSTRAP_PATH: &str = "/.openshell/channel/sandbox/bootstrap.json";
pub const RUNTIME_DESCRIPTOR_PATH: &str = "/.openshell/supervisor/runtime-descriptor.json";
pub const AUTH_BUNDLE_PATH: &str = "/.openshell/supervisor/auth.json";
pub const RESTART_METADATA_PATH: &str = "/.openshell/supervisor/restart-metadata.json";
const SOCKET_PATH: &str = "/.openshell/channel/sandbox/control.sock";
pub fn supervisor_name(id: &str) -> String {
format!("openshell-supervisor-{id}")
}
pub fn channel_volume_name(id: &str) -> String {
format!("openshell-channel-{id}")
}
fn invalid(error: impl std::fmt::Display) -> ComputeDriverError {
ComputeDriverError::Precondition(error.to_string())
}
/// Resolve policy names against the pinned workload image, never the gateway.
pub fn resolve_identity(
sandbox: &DriverSandbox,
image_id: &str,
image_user: &str,
passwd: &[u8],
group: &[u8],
) -> Result<ResolvedWorkloadIdentity, ComputeDriverError> {
let passwd = std::str::from_utf8(passwd).map_err(invalid)?;
let group = std::str::from_utf8(group).map_err(invalid)?;
let accounts: Vec<_> = passwd
.lines()
.filter_map(|line| {
let mut fields = line.split(':');
let name = fields.next()?;
fields.next()?;
Some((
name,
fields.next()?.parse::<u32>().ok()?,
fields.next()?.parse::<u32>().ok()?,
))
})
.collect();
let groups: Vec<_> = group
.lines()
.filter_map(|line| {
let mut fields = line.split(':');
let name = fields.next()?;
fields.next()?;
Some((name, fields.next()?.parse::<u32>().ok()?, fields.next()?))
})
.collect();
let request = sandbox
.spec
.as_ref()
.and_then(|spec| spec.workload_identity.as_ref());
let requested_user = request.map_or("", |identity| identity.user.trim());
let requested_group = request.map_or("", |identity| identity.group.trim());
let (image_user, image_group) = image_user.split_once(':').unwrap_or((image_user, ""));
let user = if requested_user.is_empty() {
image_user
} else {
requested_user
};
let group = if requested_group.is_empty() {
image_group
} else {
requested_group
};
let account = accounts
.iter()
.find(|(name, uid, _)| *name == user || user.parse::<u32>().ok() == Some(*uid));
let uid = user
.parse()
.ok()
.or_else(|| account.map(|(_, uid, _)| *uid))
.ok_or_else(|| invalid("configure a non-root workload user present in the pinned image"))?;
let gid = if group.is_empty() {
account.map(|(_, _, gid)| *gid)
} else {
group.parse().ok().or_else(|| {
groups
.iter()
.find(|(name, _, _)| *name == group)
.map(|(_, gid, _)| *gid)
})
}
.ok_or_else(|| {
invalid("configure an explicit workload group for a UID without an image passwd entry")
})?;
let supplemental = account.map_or_else(Vec::new, |(username, _, _)| {
groups
.iter()
.filter(|(_, id, members)| {
*id != gid && members.split(',').any(|member| member == *username)
})
.map(|(_, gid, _)| *gid)
.collect()
});
let source = if requested_user.is_empty() && requested_group.is_empty() {
"image"
} else {
"policy"
};
ResolvedWorkloadIdentity::new(uid, gid, supplemental, source.into(), image_id.into())
.map_err(invalid)
}
pub struct BootstrapArchives {
pub channel: Vec<u8>,
pub workspace: Vec<u8>,
pub supervisor: Vec<u8>,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct RestartMetadata {
pub(crate) generation: String,
pub(crate) workload_identity: ResolvedWorkloadIdentity,
pub(crate) child_env: HashMap<String, String>,
}
/// The shared volume contains only sandbox credentials. Supervisor credentials,
/// gateway authorization, and the restart copy never enter that volume.
pub fn bootstrap_archives(
sandbox_id: &str,
container_id: &str,
generation: &str,
identity: &ResolvedWorkloadIdentity,
child_env: HashMap<String, String>,
launch_authentication: &openshell_core::jwt::SandboxLaunchAuthentication,
) -> Result<BootstrapArchives, ComputeDriverError> {
launch_authentication.validate().map_err(invalid)?;
let session_id = launch_authentication.supervisor.session_id;
let tls = generate_sandbox_tls_material(session_id).map_err(invalid)?;
let resource_claims = BTreeMap::from([
("podman.container_id".into(), container_id.into()),
(
"podman.image_identity".into(),
identity.resource_digest.clone(),
),
]);
let driver_fence = DriverFenceEvidence::Podman {
container_id: container_id.into(),
network_mode: "none".into(),
unexpected_networks: Vec::new(),
};
let runtime_generation = launch_authentication
.supervisor
.runtime_generation
.to_string();
let verification_keys = launch_authentication
.verification_keys
.iter()
.map(|key| {
String::from_utf8(key.public_key_pem.clone())
.map(|public_key_pem| GatewayVerificationKey {
key_id: key.key_id.clone(),
public_key_pem,
})
.map_err(invalid)
})
.collect::<Result<Vec<_>, _>>()?;
let config = BoundaryConfig {
boundary_id: sandbox_id.into(),
generation: runtime_generation.clone(),
session_id,
session_rotation: launch_authentication.supervisor.session_rotation,
auth_epoch: launch_authentication.supervisor.auth_epoch,
gateway_id: launch_authentication.gateway_id.clone(),
verification_keys,
listener: BoundaryListener::Unix {
socket_path: PathBuf::from(SOCKET_PATH),
tls: SandboxTlsServerConfig {
certificate_chain_path: PathBuf::from("/.openshell/channel/sandbox/server.crt"),
private_key_path: PathBuf::from("/.openshell/channel/sandbox/server.key"),
},
},
resource_claims: resource_claims.clone(),
resource_claim_files: BTreeMap::new(),
workload_identity: identity.clone(),
driver_fence: driver_fence.clone(),
child_env: child_env.clone(),
};
let runtime_descriptor = SandboxRuntimeDescriptor {
boundary_id: sandbox_id.into(),
generation: runtime_generation,
session_id,
transport: SandboxTransport::Unix {
socket_path: PathBuf::from(SOCKET_PATH),
},
tls: SandboxTlsClientConfig {
server_name: tls.server_name,
trust_anchor_pem: tls.trust_anchor_pem,
},
host_gateway_ip: None,
resource_claims,
workload_identity: identity.clone(),
driver_fence,
};
// Libpod resolves the requested upload destination once for a stopped
// container. Archive entries must be relative to the selected named volume,
// not rootfs paths that the volume would shadow on container start.
let mut channel = Archive::new(identity);
channel.directory(".", 0o755, false)?;
channel.directory("sandbox", 0o711, true)?;
channel.file(
"sandbox/bootstrap.json",
&serde_json::to_vec(&config).map_err(invalid)?,
)?;
channel.file("sandbox/server.crt", tls.certificate_chain_pem.as_bytes())?;
channel.file("sandbox/server.key", tls.private_key_pem.as_bytes())?;
let channel = channel.finish()?;
let mut workspace = Archive::new(identity);
workspace.directory(".", 0o700, true)?;
let mut supervisor = Archive::new(identity);
supervisor.directory(".openshell", 0o755, false)?;
supervisor.directory(".openshell/supervisor", 0o700, true)?;
supervisor.file(
RUNTIME_DESCRIPTOR_PATH,
&serde_json::to_vec(&runtime_descriptor).map_err(invalid)?,
)?;
supervisor.file(
AUTH_BUNDLE_PATH,
&serde_json::to_vec(&launch_authentication.supervisor).map_err(invalid)?,
)?;
let restart_metadata = RestartMetadata {
generation: generation.to_string(),
workload_identity: identity.clone(),
child_env,
};
supervisor.file(
RESTART_METADATA_PATH,
&serde_json::to_vec(&restart_metadata).map_err(invalid)?,
)?;
Ok(BootstrapArchives {
channel,
workspace: workspace.finish()?,
supervisor: supervisor.finish()?,
})
}
pub fn restart_metadata_from_slice(bytes: &[u8]) -> Result<RestartMetadata, ComputeDriverError> {
serde_json::from_slice(bytes).map_err(invalid)
}
struct Archive<'a> {
builder: tar::Builder<Vec<u8>>,
identity: &'a ResolvedWorkloadIdentity,
}
impl<'a> Archive<'a> {
fn new(identity: &'a ResolvedWorkloadIdentity) -> Self {
Self {
builder: tar::Builder::new(Vec::new()),
identity,
}
}
fn directory(&mut self, path: &str, mode: u32, owned: bool) -> Result<(), ComputeDriverError> {
self.append(path, mode, owned, tar::EntryType::Directory, &[])
}
fn file(&mut self, path: &str, content: &[u8]) -> Result<(), ComputeDriverError> {
self.append(
path.trim_start_matches('/'),
0o600,
true,
tar::EntryType::Regular,
content,
)
}
fn append(
&mut self,
path: &str,
mode: u32,
owned: bool,
kind: tar::EntryType,
content: &[u8],
) -> Result<(), ComputeDriverError> {
let mut header = tar::Header::new_gnu();
header.set_entry_type(kind);
header.set_mode(mode);
header.set_uid(if owned {
u64::from(self.identity.uid)
} else {
0
});
header.set_gid(if owned {
u64::from(self.identity.gid)
} else {
0
});
header.set_size(content.len() as u64);
header.set_mtime(0);
header.set_cksum();
self.builder
.append_data(&mut header, path, content)
.map_err(invalid)
}
fn finish(self) -> Result<Vec<u8>, ComputeDriverError> {
self.builder.into_inner().map_err(invalid)
}
}
#[cfg(test)]
mod tests {
use super::*;
use openshell_core::jwt::{
CredentialEpoch, SandboxLaunchAuthentication, SecretJwt, SessionVerificationKey,
SupervisorAuthBundle,
};
fn authentication() -> SandboxLaunchAuthentication {
SandboxLaunchAuthentication {
supervisor: SupervisorAuthBundle {
session_id: openshell_core::SandboxSessionId::new(),
runtime_generation: openshell_core::sandbox_generation::SandboxGenerationId::parse(
"generation-1",
)
.unwrap(),
session_rotation: openshell_core::jwt::SessionRotation::new(1).unwrap(),
auth_epoch: CredentialEpoch::new(1).unwrap(),
gateway_token: SecretJwt::parse("gateway.token.value").unwrap(),
gateway_expires_at: i64::MAX,
sandbox_token: SecretJwt::parse("sandbox.token.value").unwrap(),
sandbox_expires_at: i64::MAX,
},
gateway_id: "gateway-test".to_string(),
verification_keys: vec![SessionVerificationKey {
key_id: "test-key".to_string(),
public_key_pem: b"public-key".to_vec(),
}],
}
}
#[test]
fn identity_uses_pinned_image_accounts_and_rejects_root() {
let sandbox = DriverSandbox::default();
let passwd = b"root:x:0:0:root:/root:/bin/sh\nagent:x:1000:1001::/home/agent:/bin/sh\n";
let groups = b"agent:x:1001:\ndata:x:2000:agent\n";
let identity =
resolve_identity(&sandbox, "sha256:pinned", "agent", passwd, groups).unwrap();
assert_eq!((identity.uid, identity.gid), (1000, 1001));
assert_eq!(identity.supplementary_gids, vec![2000]);
assert_eq!(identity.resource_digest, "sha256:pinned");
assert!(resolve_identity(&sandbox, "sha256:pinned", "root", passwd, groups).is_err());
assert!(resolve_identity(&sandbox, "sha256:pinned", "", passwd, groups).is_err());
assert!(resolve_identity(&sandbox, "sha256:pinned", "2000", passwd, groups).is_err());
}
fn files(bytes: &[u8]) -> BTreeMap<PathBuf, Vec<u8>> {
tar::Archive::new(bytes)
.entries()
.unwrap()
.filter_map(|entry| {
let mut entry = entry.unwrap();
if !entry.header().entry_type().is_file() {
return None;
}
let path = entry.path().unwrap().into_owned();
assert_eq!(entry.header().mode().unwrap(), 0o600);
assert_eq!(entry.header().uid().unwrap(), 1000);
let mut content = Vec::new();
entry.read_to_end(&mut content).unwrap();
Some((path, content))
})
.collect()
}
#[test]
fn archives_separate_supervisor_credentials_and_bind_one_channel() {
let identity = ResolvedWorkloadIdentity::new(
1000,
1001,
vec![],
"image".into(),
"sha256:image".into(),
)
.unwrap();
let authentication = authentication();
let child_env = HashMap::from([("PATH".to_string(), "/agent/bin".to_string())]);
let archives = bootstrap_archives(
"sandbox",
"container",
"generation-1",
&identity,
child_env.clone(),
&authentication,
)
.unwrap();
let workload = files(&archives.channel);
let supervisor = files(&archives.supervisor);
let mut workspace = tar::Archive::new(archives.workspace.as_slice());
let mut entries = workspace.entries().unwrap();
let root = entries.next().unwrap().unwrap();
assert_eq!(root.path().unwrap().as_ref(), std::path::Path::new("."));
assert!(root.header().entry_type().is_dir());
assert_eq!(root.header().uid().unwrap(), u64::from(identity.uid));
assert_eq!(root.header().gid().unwrap(), u64::from(identity.gid));
assert_eq!(root.header().mode().unwrap(), 0o700);
assert!(entries.next().is_none());
assert_eq!(workload.len(), 3);
assert_eq!(supervisor.len(), 3);
assert!(workload.keys().all(|path| path.starts_with("sandbox")));
assert!(
supervisor
.keys()
.all(|path| path.starts_with(".openshell/supervisor"))
);
let config: BoundaryConfig = serde_json::from_slice(
workload
.get(&PathBuf::from("sandbox/bootstrap.json"))
.unwrap(),
)
.unwrap();
let runtime_descriptor: SandboxRuntimeDescriptor = serde_json::from_slice(
supervisor
.get(&PathBuf::from(
RUNTIME_DESCRIPTOR_PATH.trim_start_matches('/'),
))
.unwrap(),
)
.unwrap();
assert_eq!(config.boundary_id, runtime_descriptor.boundary_id);
assert_eq!(config.session_id, runtime_descriptor.session_id);
assert_eq!(config.driver_fence, runtime_descriptor.driver_fence);
assert_eq!(config.workload_identity, identity);
runtime_descriptor.driver_fence.validate().unwrap();
let restart_metadata: RestartMetadata = serde_json::from_slice(
supervisor
.get(&PathBuf::from(
RESTART_METADATA_PATH.trim_start_matches('/'),
))
.unwrap(),
)
.unwrap();
assert_eq!(restart_metadata.workload_identity, identity);
assert_eq!(restart_metadata.child_env, child_env);
let restart_bytes = serde_json::to_vec(&restart_metadata).unwrap();
assert!(
!restart_bytes
.windows(b"PRIVATE KEY".len())
.any(|window| window == b"PRIVATE KEY")
);
}
}
crates/openshell-driver-podman/src/lib.rs
+1
View File
@@ -6,6 +6,7 @@ pub mod config;
pub(crate) mod container;
pub mod driver;
pub mod grpc;
mod isolation;
pub mod otel_tracing;
mod socket_discovery;
#[cfg(test)]
crates/openshell-driver-podman/src/main.rs
+8 -1
View File
@@ -101,7 +101,11 @@ struct Args {
)]
health_check_interval_secs: Option<NonZeroU64>,
/// OCI image containing the openshell-sandbox supervisor binary.
/// OCI image containing the `openshell-sandbox` runtime binary.
#[arg(long, env = "OPENSHELL_SANDBOX_RUNTIME_IMAGE")]
sandbox_runtime_image: Option<String>,
/// OCI image containing the `openshell-supervisor` control binary.
#[arg(long, env = "OPENSHELL_SUPERVISOR_IMAGE")]
supervisor_image: Option<String>,
@@ -207,6 +211,9 @@ async fn main() -> Result<()> {
ssh_socket_path: args.sandbox_ssh_socket_path,
network_name: args.network_name,
stop_timeout_secs: args.stop_timeout,
sandbox_runtime_image: args
.sandbox_runtime_image
.unwrap_or_else(openshell_core::config::default_sandbox_runtime_image),
supervisor_image: args
.supervisor_image
.unwrap_or_else(openshell_core::config::default_supervisor_image),
crates/openshell-driver-podman/src/test_utils.rs
+23 -5
View File
@@ -3,7 +3,7 @@
//! Shared test helpers for openshell-driver-podman unit tests.
use http_body_util::Full;
use http_body_util::{BodyExt as _, Full};
use hyper::StatusCode;
use hyper::body::Bytes;
use hyper::server::conn::http1;
@@ -20,16 +20,18 @@ use tokio::net::UnixListener;
#[derive(Clone)]
pub struct StubResponse {
pub status: StatusCode,
pub body: String,
pub body: Bytes,
pub delay: Duration,
pub archive_members: Option<Vec<PathBuf>>,
}
impl StubResponse {
pub fn new(status: StatusCode, body: impl Into<String>) -> Self {
pub fn new(status: StatusCode, body: impl Into<Bytes>) -> Self {
Self {
status,
body: body.into(),
delay: Duration::ZERO,
archive_members: None,
}
}
@@ -37,6 +39,11 @@ impl StubResponse {
self.delay = delay;
self
}
pub fn with_archive_members(mut self, members: &[&str]) -> Self {
self.archive_members = Some(members.iter().map(PathBuf::from).collect());
self
}
}
/// Generate a unique Unix socket path for a test.
@@ -88,7 +95,7 @@ pub fn spawn_podman_stub(
let result = http1::Builder::new()
.serve_connection(
TokioIo::new(stream),
service_fn(move |req| {
service_fn(move |req: hyper::Request<hyper::body::Incoming>| {
let log = log.clone();
let queue = queue.clone();
async move {
@@ -104,11 +111,22 @@ pub fn spawn_podman_stub(
.expect("response queue lock should not be poisoned")
.pop_front()
.expect("stub response should exist");
if let Some(expected_members) = &response.archive_members {
assert_eq!(req.method(), hyper::Method::PUT);
let body = req.into_body().collect().await.unwrap().to_bytes();
let mut archive = tar::Archive::new(body.as_ref());
let members: Vec<_> = archive
.entries()
.unwrap()
.map(|entry| entry.unwrap().path().unwrap().into_owned())
.collect();
assert_eq!(&members, expected_members);
}
tokio::time::sleep(response.delay).await;
Ok::<_, Infallible>(
hyper::Response::builder()
.status(response.status)
.body(Full::new(Bytes::from(response.body)))
.body(Full::new(response.body))
.expect("stub response should build"),
)
}
crates/openshell-driver-podman/src/watcher.rs
+137 -3
View File
@@ -139,14 +139,16 @@ pub async fn start_watch(
let mut event_rx = client.events_stream(LABEL_MANAGED_FILTER).await?;
// 2. List existing containers for initial state sync.
let existing = client.list_containers(&[LABEL_MANAGED_FILTER]).await?;
let existing = client
.list_containers(&[LABEL_MANAGED_FILTER, crate::isolation::WORKLOAD_FILTER])
.await?;
for entry in &existing {
// For running containers, use inspect to get full state including
// health check status — matching the same condition derivation used
// for live events.
if entry.state == "running" {
match client.inspect_container(&entry.id).await {
match inspect_workload(&client, &entry.id).await {
Ok(inspect) => {
if let Some(sandbox) = driver_sandbox_from_inspect(&inspect) {
if tx.send(Ok(sandbox_event(sandbox))).await.is_err() {
@@ -246,11 +248,34 @@ async fn map_podman_event(
return None;
}
if event
.actor
.attributes
.get(crate::isolation::LABEL_ROLE)
.is_some_and(|role| role == "supervisor")
{
let id_filter = format!("{LABEL_SANDBOX_ID}={sandbox_id}");
let workloads = client
.list_containers(&[
LABEL_MANAGED_FILTER,
&id_filter,
crate::isolation::WORKLOAD_FILTER,
])
.await
.ok()?;
let workload = workloads.first()?;
return inspect_workload(client, &workload.id)
.await
.ok()
.and_then(|inspect| driver_sandbox_from_inspect(&inspect))
.map(sandbox_event);
}
match event.action.as_str() {
"remove" => Some(deleted_event(sandbox_id.clone())),
"create" | "start" | "stop" | "die" | "health_status" => {
// Inspect the container to get current state.
match client.inspect_container(container_id).await {
match inspect_workload(client, container_id).await {
Ok(inspect) => {
if lifecycle_event_fences.matches_previous_exit(
event,
@@ -327,6 +352,54 @@ async fn map_podman_event(
}
}
/// A workload is ready only when its independent supervisor is healthy. This
/// check runs both on watch reconciliation and on events, and contains a lost
/// supervisor even when the gateway missed the original exit event.
pub async fn inspect_workload(
client: &PodmanClient,
id: &str,
) -> Result<ContainerInspect, PodmanApiError> {
let mut workload = client.inspect_container(id).await?;
if workload
.config
.labels
.get(crate::isolation::LABEL_ROLE)
.is_none_or(|role| role != "sandbox")
{
return Ok(workload);
}
let Some(sandbox_id) = workload.config.labels.get(LABEL_SANDBOX_ID) else {
return Ok(workload);
};
let supervisor = client
.inspect_container(&crate::isolation::supervisor_name(sandbox_id))
.await;
if workload.state.running {
match supervisor {
Ok(supervisor) if supervisor.state.running => {
workload.state.health = supervisor.state.health;
}
Ok(supervisor)
if supervisor.state.status == "configured"
|| supervisor.state.status == "created" =>
{
workload.state.health = Some(HealthState {
status: "starting".into(),
});
}
// Both containers exist before initial start. A missing or exited
// companion therefore requires containment, including after a
// gateway restart that missed the original Podman exit event.
Ok(_) | Err(PodmanApiError::NotFound(_)) => {
client.stop_container(&workload.id, 0).await?;
workload = client.inspect_container(&workload.id).await?;
}
Err(error) => return Err(error),
}
}
Ok(workload)
}
/// Construct a `DriverSandbox` from common fields.
///
/// Centralises the boilerplate that every event/inspect/list path shares:
@@ -353,6 +426,7 @@ fn build_driver_sandbox(
sandbox_fd: String::new(),
conditions: vec![condition],
deleting,
..Default::default()
}),
workspace,
}
@@ -509,6 +583,65 @@ fn condition_from_state(state: &ContainerState) -> DriverCondition {
mod tests {
use super::*;
#[tokio::test]
async fn missing_supervisor_stops_workload_during_reconciliation() {
use crate::test_utils::{StubResponse, spawn_podman_stub};
use hyper::StatusCode;
let (path, requests, handle) = spawn_podman_stub(
"lost-supervisor",
vec![
StubResponse::new(
StatusCode::OK,
r#"{"Id":"workload","Name":"workload","State":{"Status":"running","Running":true},"Config":{"Labels":{"openshell.ai/sandbox-id":"test","openshell.io/isolation-role":"sandbox"}}}"#,
),
StubResponse::new(StatusCode::NOT_FOUND, "missing companion"),
StubResponse::new(StatusCode::NO_CONTENT, ""),
StubResponse::new(
StatusCode::OK,
r#"{"Id":"workload","Name":"workload","State":{"Status":"exited","Running":false},"Config":{}}"#,
),
],
);
let client = PodmanClient::new(path.clone());
let inspected = inspect_workload(&client, "workload").await.unwrap();
assert!(!inspected.state.running);
handle.await.unwrap();
assert!(
requests
.lock()
.unwrap()
.iter()
.any(|request| request.ends_with("/libpod/containers/workload/stop?timeout=0"))
);
let _ = std::fs::remove_file(path);
}
#[tokio::test]
async fn created_supervisor_keeps_bootstrapping_workload_starting() {
use crate::test_utils::{StubResponse, spawn_podman_stub};
use hyper::StatusCode;
let (path, requests, handle) = spawn_podman_stub(
"starting-supervisor",
vec![
StubResponse::new(
StatusCode::OK,
r#"{"Id":"workload","Name":"workload","State":{"Status":"running","Running":true},"Config":{"Labels":{"openshell.ai/sandbox-id":"test","openshell.io/isolation-role":"sandbox"}}}"#,
),
StubResponse::new(
StatusCode::OK,
r#"{"Id":"supervisor","Name":"supervisor","State":{"Status":"configured","Running":false},"Config":{}}"#,
),
],
);
let client = PodmanClient::new(path.clone());
let inspected = inspect_workload(&client, "workload").await.unwrap();
assert!(inspected.state.running);
assert_eq!(inspected.state.health.unwrap().status, "starting");
handle.await.unwrap();
assert_eq!(requests.lock().unwrap().len(), 2);
let _ = std::fs::remove_file(path);
}
fn podman_event(action: &str, sandbox_id: &str, time_nano: i64) -> PodmanEvent {
PodmanEvent {
event_type: "container".to_string(),
@@ -761,6 +894,7 @@ mod tests {
sandbox_fd: String::new(),
conditions: vec![condition],
deleting: false,
..Default::default()
}),
workspace: String::new(),
};
crates/openshell-driver-vault/Cargo.toml
+3
View File
@@ -30,9 +30,12 @@ toml = { workspace = true }
tonic = { workspace = true }
tracing = { workspace = true }
tracing-subscriber = { workspace = true }
url = { workspace = true }
[dev-dependencies]
rcgen = { workspace = true }
tempfile = "3"
tokio-rustls = { workspace = true }
wiremock = "0.6"
[lints]
crates/openshell-driver-vault/src/lib.rs
+383 -9
View File
@@ -17,7 +17,7 @@ use openshell_core::proto::credentials::v1::{
credential_driver_server::CredentialDriver,
};
use openshell_core::{Error, Result as CoreResult};
use reqwest::{StatusCode, Url};
use reqwest::{Certificate, StatusCode, Url};
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use tonic::{Request, Response, Status};
@@ -58,6 +58,7 @@ impl CredentialDriverService {
#[derive(Debug, Clone, PartialEq, Eq)]
struct VaultDriverSettings {
address: Url,
ca_bundle: Option<PathBuf>,
mount: String,
kv_version: KvVersion,
auth: VaultAuthSettings,
@@ -85,6 +86,7 @@ enum KvVersion {
#[serde(default, deny_unknown_fields)]
struct VaultDriverConfig {
address: Option<String>,
ca_bundle: Option<PathBuf>,
mount: Option<String>,
kv_version: Option<String>,
auth_method: Option<String>,
@@ -127,14 +129,43 @@ impl VaultCredentialDriver {
let settings = VaultDriverSettings::from_table(config)?;
let timeout_secs = timeout_secs(config)?;
let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();
let client = reqwest::Client::builder()
let mut client_builder = reqwest::Client::builder()
.timeout(Duration::from_secs(timeout_secs))
.build()
.map_err(|err| {
// System HTTP proxies can turn an otherwise-local plaintext request
// into a network request. Vault credentials must always go directly
// to the configured backend.
.no_proxy()
// Vault API requests carry bootstrap identity, backend tokens, and
// provider secrets. Never replay them to a redirect target.
.redirect(reqwest::redirect::Policy::none());
if let Some(ca_bundle) = settings.ca_bundle.as_deref() {
let pem = std::fs::read(ca_bundle).map_err(|err| {
Error::config(format!(
"failed to configure vault credential driver: {err}"
"failed to read Vault CA bundle '{}': {err}",
ca_bundle.display()
))
})?;
let certificates = Certificate::from_pem_bundle(&pem).map_err(|err| {
Error::config(format!(
"failed to parse Vault CA bundle '{}': {err}",
ca_bundle.display()
))
})?;
if certificates.is_empty() {
return Err(Error::config(format!(
"Vault CA bundle '{}' contains no certificates",
ca_bundle.display()
)));
}
for certificate in certificates {
client_builder = client_builder.add_root_certificate(certificate);
}
}
let client = client_builder.build().map_err(|err| {
Error::config(format!(
"failed to configure vault credential driver: {err}"
))
})?;
Ok(Self {
client,
settings,
@@ -537,6 +568,11 @@ impl VaultDriverSettings {
Error::config("[openshell.credential_drivers.vault] address is required")
})
.and_then(vault_address)?;
if config.ca_bundle.is_some() && address.scheme() != "https" {
return Err(Error::config(
"[openshell.credential_drivers.vault] ca_bundle requires an https address",
));
}
let mount = config
.mount
.as_deref()
@@ -601,6 +637,7 @@ impl VaultDriverSettings {
Ok(Self {
address,
ca_bundle: config.ca_bundle,
mount,
kv_version,
auth,
@@ -630,6 +667,16 @@ fn vault_address(value: &str) -> CoreResult<Url> {
"[openshell.credential_drivers.vault] address must use http or https",
));
}
if url.host().is_none() {
return Err(Error::config(
"[openshell.credential_drivers.vault] address must include a host",
));
}
if url.scheme() == "http" && !vault_host_is_loopback(&url) {
return Err(Error::config(
"[openshell.credential_drivers.vault] address must use https unless the host is loopback",
));
}
if !url.username().is_empty() || url.password().is_some() {
return Err(Error::config(
"[openshell.credential_drivers.vault] address must not include credentials",
@@ -647,6 +694,15 @@ fn vault_address(value: &str) -> CoreResult<Url> {
Ok(url)
}
fn vault_host_is_loopback(url: &Url) -> bool {
match url.host() {
Some(url::Host::Ipv4(address)) => address.is_loopback(),
Some(url::Host::Ipv6(address)) => address.is_loopback(),
Some(url::Host::Domain(domain)) => domain.eq_ignore_ascii_case("localhost"),
None => false,
}
}
fn timeout_secs(table: &toml::Table) -> CoreResult<u64> {
let Some(value) = table.get("timeout_secs") else {
return Ok(DEFAULT_TIMEOUT_SECS);
@@ -915,7 +971,12 @@ fn extract_secret_value(
#[cfg(test)]
mod tests {
use std::net::SocketAddr;
use std::sync::Arc;
use openshell_core::proto::CredentialHandle;
use rcgen::{CertificateParams, IsCa, KeyPair};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tonic::Code;
use wiremock::matchers::{body_string_contains, header, method, path};
use wiremock::{Mock, MockServer, ResponseTemplate};
@@ -943,12 +1004,78 @@ mod tests {
file
}
fn test_ca() -> (rcgen::Certificate, KeyPair) {
let mut params = CertificateParams::new(Vec::<String>::new()).unwrap();
params.is_ca = IsCa::Ca(rcgen::BasicConstraints::Unconstrained);
let key = KeyPair::generate().unwrap();
let certificate = params.self_signed(&key).unwrap();
(certificate, key)
}
async fn start_tls_server(response: String) -> (SocketAddr, String) {
let (ca_certificate, ca_key) = test_ca();
let server_key = KeyPair::generate().unwrap();
let server_certificate = CertificateParams::new(vec!["localhost".to_string()])
.unwrap()
.signed_by(&server_key, &ca_certificate, &ca_key)
.unwrap();
let server_config = rustls::ServerConfig::builder()
.with_no_client_auth()
.with_single_cert(
vec![server_certificate.der().clone()],
rustls::pki_types::PrivateKeyDer::try_from(server_key.serialize_der()).unwrap(),
)
.unwrap();
let acceptor = tokio_rustls::TlsAcceptor::from(Arc::new(server_config));
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
tokio::spawn(async move {
let Ok((socket, _)) = listener.accept().await else {
return;
};
let Ok(mut tls) = acceptor.accept(socket).await else {
return;
};
let mut request = vec![0_u8; 4096];
let mut used = 0;
loop {
let Ok(read) = tls.read(&mut request[used..]).await else {
return;
};
used += read;
if read == 0 || request[..used].windows(4).any(|bytes| bytes == b"\r\n\r\n") {
break;
}
if used == request.len() {
return;
}
}
let _ = tls.write_all(response.as_bytes()).await;
let _ = tls.flush().await;
});
(address, ca_certificate.pem())
}
fn ca_bundle_file(pem: &str) -> tempfile::NamedTempFile {
let file = tempfile::NamedTempFile::new().unwrap();
std::fs::write(file.path(), pem).unwrap();
file
}
fn test_reference() -> VaultSecretReference {
VaultSecretReference {
api_path: "secret/data/test".to_string(),
key: STORED_VALUE_KEY.to_string(),
kv_version: KvVersion::V2,
}
}
#[test]
fn settings_parse_kubernetes_auth() {
let settings = VaultDriverSettings::from_table(&table(&[
(
"address",
toml::Value::String("http://vault:8200".to_string()),
toml::Value::String("https://vault:8200".to_string()),
),
("mount", toml::Value::String("team-secret".to_string())),
("kv_version", toml::Value::String("1".to_string())),
@@ -970,7 +1097,7 @@ mod tests {
let settings = VaultDriverSettings::from_table(&table(&[
(
"address",
toml::Value::String("http://vault:8200".to_string()),
toml::Value::String("https://vault:8200".to_string()),
),
("auth_method", toml::Value::String("token_file".to_string())),
(
@@ -989,7 +1116,7 @@ mod tests {
let err = VaultDriverSettings::from_table(&table(&[
(
"address",
toml::Value::String("http://vault:8200".to_string()),
toml::Value::String("https://vault:8200".to_string()),
),
("auth_method", toml::Value::String("token_file".to_string())),
(
@@ -1008,7 +1135,7 @@ mod tests {
let err = VaultDriverSettings::from_table(&table(&[
(
"address",
toml::Value::String("http://vault:8200".to_string()),
toml::Value::String("https://vault:8200".to_string()),
),
("auth_method", toml::Value::String("token_file".to_string())),
]))
@@ -1017,6 +1144,253 @@ mod tests {
assert!(err.to_string().contains("token_path is required"));
}
#[test]
fn settings_reject_non_loopback_http_address() {
let err = vault_address("http://vault.vault.svc.cluster.local:8200").unwrap_err();
assert!(err.to_string().contains("must use https"), "{err}");
}
#[test]
fn settings_allow_loopback_http_addresses() {
for address in [
"http://127.0.0.1:8200",
"http://[::1]:8200",
"http://localhost:8200",
] {
vault_address(address).unwrap();
}
}
#[test]
fn settings_reject_ca_bundle_for_http() {
let err = VaultDriverSettings::from_table(&table(&[
(
"address",
toml::Value::String("http://127.0.0.1:8200".to_string()),
),
(
"ca_bundle",
toml::Value::String("/etc/vault/ca.pem".to_string()),
),
("auth_method", toml::Value::String("token_file".to_string())),
(
"token_path",
toml::Value::String("/run/secrets/vault-token".to_string()),
),
]))
.unwrap_err();
assert!(err.to_string().contains("ca_bundle requires an https"));
}
#[tokio::test]
async fn https_uses_custom_ca_bundle() {
let body = r#"{"data":{"data":{"value":"protected-secret"}}}"#;
let response = format!(
"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
body.len()
);
let (address, ca_pem) = start_tls_server(response).await;
let ca_bundle = ca_bundle_file(&ca_pem);
let token = token_file("dev-token");
let driver = VaultCredentialDriver::from_config(&table(&[
(
"address",
toml::Value::String(format!("https://localhost:{}/", address.port())),
),
(
"ca_bundle",
toml::Value::String(ca_bundle.path().display().to_string()),
),
("auth_method", toml::Value::String("token_file".to_string())),
(
"token_path",
toml::Value::String(token.path().display().to_string()),
),
]))
.unwrap();
let value = driver
.resolve_secret_value(&test_reference(), "dev-token")
.await
.unwrap();
assert_eq!(value, "protected-secret");
}
#[tokio::test]
async fn https_rejects_invalid_ca() {
let (address, _) = start_tls_server(String::new()).await;
let (untrusted_ca, _) = test_ca();
let ca_bundle = ca_bundle_file(&untrusted_ca.pem());
let token = token_file("dev-token");
let driver = VaultCredentialDriver::from_config(&table(&[
(
"address",
toml::Value::String(format!("https://localhost:{}/", address.port())),
),
(
"ca_bundle",
toml::Value::String(ca_bundle.path().display().to_string()),
),
("auth_method", toml::Value::String("token_file".to_string())),
(
"token_path",
toml::Value::String(token.path().display().to_string()),
),
]))
.unwrap();
let err = driver
.resolve_secret_value(&test_reference(), "dev-token")
.await
.unwrap_err();
assert_eq!(err.code(), Code::Unavailable);
}
#[tokio::test]
async fn https_preserves_hostname_verification() {
let (address, ca_pem) = start_tls_server(String::new()).await;
let ca_bundle = ca_bundle_file(&ca_pem);
let token = token_file("dev-token");
let driver = VaultCredentialDriver::from_config(&table(&[
(
"address",
toml::Value::String(format!("https://127.0.0.1:{}/", address.port())),
),
(
"ca_bundle",
toml::Value::String(ca_bundle.path().display().to_string()),
),
("auth_method", toml::Value::String("token_file".to_string())),
(
"token_path",
toml::Value::String(token.path().display().to_string()),
),
]))
.unwrap();
let err = driver
.resolve_secret_value(&test_reference(), "dev-token")
.await
.unwrap_err();
assert_eq!(err.code(), Code::Unavailable);
}
#[tokio::test]
async fn https_does_not_follow_redirect_downgrade() {
let downgrade_target = MockServer::start().await;
let location = format!("{}/stolen", downgrade_target.uri());
let response = format!(
"HTTP/1.1 307 Temporary Redirect\r\nLocation: {location}\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
);
let (address, ca_pem) = start_tls_server(response).await;
let ca_bundle = ca_bundle_file(&ca_pem);
let token = token_file("dev-token");
let driver = VaultCredentialDriver::from_config(&table(&[
(
"address",
toml::Value::String(format!("https://localhost:{}/", address.port())),
),
(
"ca_bundle",
toml::Value::String(ca_bundle.path().display().to_string()),
),
("auth_method", toml::Value::String("token_file".to_string())),
(
"token_path",
toml::Value::String(token.path().display().to_string()),
),
]))
.unwrap();
driver
.login_kubernetes("openshell-gateway", "kubernetes", "gateway-jwt")
.await
.unwrap_err();
assert!(
downgrade_target
.received_requests()
.await
.unwrap()
.is_empty()
);
}
#[test]
fn loopback_http_bypasses_system_proxy() {
let proxy_listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
let proxy_address = proxy_listener.local_addr().unwrap();
drop(proxy_listener);
let output = std::process::Command::new(std::env::current_exe().unwrap())
.args([
"--exact",
"tests::loopback_http_bypasses_system_proxy_child",
"--ignored",
"--nocapture",
])
.env("OPENSHELL_VAULT_PROXY_TEST_CHILD", "1")
.env("HTTP_PROXY", format!("http://{proxy_address}"))
.env("http_proxy", format!("http://{proxy_address}"))
.env_remove("NO_PROXY")
.env_remove("no_proxy")
.env_remove("ALL_PROXY")
.env_remove("all_proxy")
.env_remove("REQUEST_METHOD")
.output()
.unwrap();
assert!(
output.status.success(),
"child test failed:\nstdout:\n{}\nstderr:\n{}",
String::from_utf8_lossy(&output.stdout),
String::from_utf8_lossy(&output.stderr)
);
}
#[tokio::test]
#[ignore = "run by loopback_http_bypasses_system_proxy in an isolated process"]
async fn loopback_http_bypasses_system_proxy_child() {
if std::env::var_os("OPENSHELL_VAULT_PROXY_TEST_CHILD").is_none() {
return;
}
let mock_server = MockServer::start().await;
Mock::given(method("GET"))
.and(path("/v1/secret/data/test"))
.and(header("x-vault-token", "dev-token"))
.respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({
"data": {"data": {"value": "protected-secret"}}
})))
.mount(&mock_server)
.await;
let token = token_file("dev-token");
let driver = VaultCredentialDriver::from_config(&table(&[
(
"address",
toml::Value::String(format!("{}/", mock_server.uri())),
),
("auth_method", toml::Value::String("token_file".to_string())),
(
"token_path",
toml::Value::String(token.path().display().to_string()),
),
]))
.unwrap();
let value = driver
.resolve_secret_value(&test_reference(), "dev-token")
.await
.unwrap();
assert_eq!(value, "protected-secret");
}
#[test]
fn api_path_builds_kv2_api_path_from_logical_path() {
assert_eq!(
crates/openshell-driver-vault/src/main.rs
+4
View File
@@ -30,6 +30,9 @@ struct Args {
#[arg(long, env = "OPENSHELL_VAULT_ADDRESS")]
address: Option<String>,
#[arg(long, env = "OPENSHELL_VAULT_CA_BUNDLE")]
ca_bundle: Option<PathBuf>,
#[arg(long, env = "OPENSHELL_VAULT_MOUNT")]
mount: Option<String>,
@@ -85,6 +88,7 @@ async fn main() -> Result<()> {
fn driver_config(args: &Args) -> toml::Table {
let mut config = toml::Table::new();
insert_string(&mut config, "address", args.address.as_ref());
insert_path(&mut config, "ca_bundle", args.ca_bundle.as_ref());
insert_string(&mut config, "mount", args.mount.as_ref());
insert_string(&mut config, "kv_version", args.kv_version.as_ref());
insert_string(&mut config, "auth_method", args.auth_method.as_ref());
crates/openshell-driver-vm/Cargo.toml
+74 -31
View File
@@ -17,45 +17,88 @@ path = "src/lib.rs"
[[bin]]
name = "openshell-driver-vm"
path = "src/main.rs"
required-features = ["compute-driver"]
[[bin]]
name = "openshell-vm-init"
path = "src/bin/openshell-vm-init.rs"
[dependencies]
openshell-core = { path = "../openshell-core", default-features = false }
openshell-otel = { path = "../openshell-otel" }
openshell-policy = { path = "../openshell-policy" }
openshell-driver-podman = { path = "../openshell-driver-podman" }
openshell-vfio = { path = "../openshell-vfio" }
openshell-isolation-interface = { path = "../openshell-isolation-interface" }
openshell-sandbox-backend = { path = "../openshell-sandbox-backend" }
openshell-otel = { path = "../openshell-otel", optional = true }
openshell-policy = { path = "../openshell-policy", optional = true }
openshell-driver-podman = { path = "../openshell-driver-podman", optional = true }
openshell-vfio = { path = "../openshell-vfio", optional = true }
bollard = { version = "0.20", features = ["ssh"] }
base64 = { workspace = true, optional = true }
bollard = { version = "0.20", features = ["ssh"], optional = true }
tokio = { workspace = true }
tonic = { workspace = true, features = ["transport"] }
tower-http = { workspace = true }
http = { workspace = true }
prost = { workspace = true }
prost-types = { workspace = true }
futures = { workspace = true }
tokio-stream = { workspace = true, features = ["net"] }
nix = { workspace = true }
clap = { workspace = true }
tracing = { workspace = true }
tracing-subscriber = { workspace = true }
opentelemetry = { workspace = true }
opentelemetry_sdk = { workspace = true }
tracing-opentelemetry = { workspace = true }
miette = { workspace = true }
url = { workspace = true }
tonic = { workspace = true, features = ["transport"], optional = true }
tower-http = { workspace = true, optional = true }
http = { workspace = true, optional = true }
prost = { workspace = true, optional = true }
prost-types = { workspace = true, optional = true }
futures = { workspace = true, optional = true }
tokio-stream = { workspace = true, features = ["net"], optional = true }
nix = { workspace = true, optional = true }
clap = { workspace = true, optional = true }
tracing = { workspace = true, optional = true }
tracing-subscriber = { workspace = true, optional = true }
opentelemetry = { workspace = true, optional = true }
opentelemetry_sdk = { workspace = true, optional = true }
tracing-opentelemetry = { workspace = true, optional = true }
miette = { workspace = true, optional = true }
rand = { workspace = true, optional = true }
url = { workspace = true, optional = true }
serde = { workspace = true }
serde_json = { workspace = true }
oci-client = "0.16"
oci-client = { version = "0.16", optional = true }
libc = "0.2"
rustix = { workspace = true }
libloading = "0.8"
tar = "0.4"
flate2 = "1"
sha2 = "0.10"
zstd = "0.13"
rustix = { workspace = true, optional = true }
libloading = { version = "0.8", optional = true }
tar = { version = "0.4", optional = true }
flate2 = { version = "1", optional = true }
sha2 = { version = "0.10", optional = true }
zstd = { version = "0.13", optional = true }
[features]
default = ["telemetry"]
default = ["compute-driver", "telemetry"]
## Build the standalone compute driver and its host runtime implementation.
compute-driver = [
"dep:base64",
"dep:bollard",
"dep:clap",
"dep:flate2",
"dep:futures",
"dep:http",
"dep:libloading",
"dep:miette",
"dep:nix",
"dep:oci-client",
"dep:openshell-otel",
"dep:openshell-policy",
"dep:openshell-driver-podman",
"dep:openshell-vfio",
"dep:opentelemetry",
"dep:opentelemetry_sdk",
"dep:polling",
"dep:prost",
"dep:prost-types",
"dep:rand",
"dep:rustix",
"dep:sha2",
"dep:tar",
"dep:tokio-stream",
"dep:tonic",
"dep:tower-http",
"dep:tracing",
"dep:tracing-opentelemetry",
"dep:tracing-subscriber",
"dep:url",
"dep:zstd",
]
## Compile in telemetry support (forwards to openshell-core/telemetry). On by
## default; build with `--no-default-features` for a telemetry-free VM driver
## that reports telemetry disabled to the sandboxes it launches.
@@ -68,7 +111,7 @@ telemetry = ["openshell-core/telemetry"]
## enabling it alongside `telemetry` is a compile error rather than a silent
## telemetry-on build. Kept in sync with `default` by
## `rust:verify:defaults-without-telemetry`.
defaults-without-telemetry = []
defaults-without-telemetry = ["compute-driver"]
[dev-dependencies]
openshell-otel-test-support = { path = "../openshell-otel-test-support" }
@@ -82,7 +125,7 @@ opentelemetry_sdk = { workspace = true, features = ["testing"] }
# nix::sys::prctl::set_pdeathsig there keeps the Linux path a single
# syscall with no helper thread.
[target.'cfg(any(target_os = "macos", target_os = "ios", target_os = "freebsd", target_os = "netbsd", target_os = "openbsd", target_os = "dragonfly"))'.dependencies]
polling = "3.11"
polling = { version = "3.11", optional = true }
[lints]
workspace = true
crates/openshell-driver-vm/README.md
+55 -61
View File
@@ -1,33 +1,40 @@
# openshell-driver-vm
> Status: Experimental. The VM compute driver is under active development and the interface still has VM-specific plumbing that will be generalized.
> Status: Experimental. The VM compute driver is under active development.
Standalone libkrun-backed [`ComputeDriver`](../../proto/compute_driver.proto) for OpenShell. The gateway spawns this binary as a subprocess, talks to it over a Unix domain socket with the `openshell.compute.v1.ComputeDriver` gRPC surface, and lets it manage per-sandbox microVMs. The runtime (libkrun + libkrunfw + gvproxy), guest OCI unpacker, and sandbox supervisor are embedded directly in the binary; each sandbox boots from a cached immutable bootstrap ext4 root disk plus a per-sandbox writable overlay disk. When the requested sandbox image differs from the bootstrap image, the driver prepares a read-only image ext4 disk inside a bootstrap VM and mounts that unpacked rootfs as the sandbox lowerdir.
Standalone libkrun-backed [`ComputeDriver`](../../proto/compute_driver.proto) for OpenShell. The gateway spawns this binary as a subprocess and talks to it over the `openshell.compute.v1.ComputeDriver` Unix-socket surface. `openshell-supervisor` runs as a native host process, while `openshell-sandbox` runs as capability-free PID 1 inside each microVM and applies guest-local isolation over virtio-vsock.
The driver embeds libkrun, libkrunfw, the guest OCI unpacker, the portable guest sandbox, and the custom kernel runtime. Each sandbox boots from a cached immutable bootstrap ext4 root disk plus a per-sandbox writable overlay disk. When the requested sandbox image differs from the bootstrap image, the driver prepares a read-only image ext4 disk inside a bootstrap VM and mounts that unpacked rootfs as the sandbox lowerdir.
## How it fits together
```mermaid
flowchart LR
subgraph host["Host process"]
subgraph host["Host"]
gateway["openshell-gateway<br/>(vm::spawn)"]
driver["openshell-driver-vm<br/>├── libkrun (VM)<br/>├── gvproxy (net)<br/>└── openshell-sandbox.zst"]
driver["openshell-driver-vm<br/>libkrun"]
supervisor["openshell-supervisor<br/>host policy supervisor"]
gateway <-->|"gRPC over UDS<br/>compute-driver.sock"| driver
supervisor <-->|"authenticated gRPC<br/>policy + relay"| gateway
end
subgraph guest["Per-sandbox microVM"]
init["/srv/openshell-vm-<br/>sandbox-init.sh"]
supervisor["/opt/openshell/bin/<br/>openshell-sandbox<br/>(PID 1)"]
init --> supervisor
sandbox["openshell-sandbox<br/>capability-free guest PID 1"]
workload["sandbox workload"]
init --> sandbox --> workload
end
driver -->|"CreateSandbox<br/>boots via libkrun"| guest
supervisor -.->|"gRPC callback<br/>--grpc-endpoint"| gateway
supervisor <-->|"mutual TLS RFC 0012<br/>over virtio-vsock"| sandbox
client["openshell-cli"] -->|"SSH proxy<br/>127.0.0.1:&lt;port&gt;"| supervisor
client["openshell-cli"] -->|"connect / exec / forward"| gateway
client -->|"CreateSandbox / Watch"| gateway
```
Sandbox guests execute `/opt/openshell/bin/openshell-sandbox` as PID 1 inside the VM. gvproxy exposes a single inbound SSH port (`host:<allocated>` → `guest:2222`) and provides virtio-net egress.
The supervisor owns gateway credentials, admitted policy, provider resolution, middleware, the network proxy, and relay registration. The sandbox receives no gateway JWT. Each VM generation receives distinct sandbox and supervisor channel keys; the guest consumes and unlinks its private bootstrap files before launching the workload.
VM-specific RFC 0012 code under `src/isolation/` only chooses the vsock transport and binds immutable VM generation and image claims into the protected guest config and host descriptor. Lifecycle, authentication, process control, binary identity, forwarding, and streaming come from `openshell-isolation-interface` and `openshell-sandbox`.
## Quick start (recommended)
@@ -35,7 +42,7 @@ Sandbox guests execute `/opt/openshell/bin/openshell-sandbox` as PID 1 inside th
mise run gateway:vm
```
First run takes a few minutes while `mise run vm:setup` stages libkrun/libkrunfw/gvproxy/umoci and `mise run vm:supervisor` builds the bundled guest supervisor. Subsequent runs are cached.
First run takes a few minutes while `mise run vm:setup` stages libkrun/libkrunfw/umoci and `mise run vm:supervisor` builds the portable Linux guest sandbox plus its small static guest-init helper. The development task also builds the native host supervisor. Subsequent runs are cached.
By default `mise run gateway:vm`:
@@ -96,13 +103,13 @@ rm -rf "${XDG_CONFIG_HOME:-$HOME/.config}/openshell/gateways/vm-dev"
If you want to drive the launch yourself instead of using `mise run gateway:vm` (i.e. `tasks/scripts/gateway-vm.sh`):
```shell
# 1. Stage runtime artifacts + supervisor bundle into target/vm-runtime-compressed/
# 1. Stage runtime artifacts + guest sandbox into target/vm-runtime-compressed/
mise run vm:setup
mise run vm:supervisor # if openshell-sandbox.zst is not already present
mise run vm:supervisor # builds the Linux guest sandbox and static guest-init helper
# 2. Build both binaries with the staged artifacts embedded
# 2. Build gateway, native host supervisor, and driver
OPENSHELL_VM_RUNTIME_COMPRESSED_DIR=$PWD/target/vm-runtime-compressed \
cargo build -p openshell-gateway -p openshell-driver-vm
cargo build -p openshell-gateway -p openshell-supervisor -p openshell-driver-vm
# 3. macOS only: codesign the driver for Hypervisor.framework
codesign \
@@ -121,8 +128,7 @@ disable_tls = true
[openshell.drivers.vm]
default_image = "<compatible-image>"
# Optional override; the gateway derives host.openshell.internal:18081 when omitted.
grpc_endpoint = "http://host.openshell.internal:18081"
grpc_endpoint = "http://127.0.0.1:18081"
driver_dir = "$PWD/target/debug"
state_dir = "/tmp/openshell-vm-driver-$USER-vm-dev"
EOF
@@ -143,8 +149,8 @@ Select the VM driver with `--compute-driver vm`, `OPENSHELL_COMPUTE_DRIVER=vm`,
| Configuration key | Default | Purpose |
|---|---|---|
| `grpc_endpoint` | topology-derived | Optional override for the URL the sandbox guest dials to reach the gateway. The gateway derives `http(s)://host.openshell.internal:<gateway-port>` when absent. Use `host.containers.internal`, `host.docker.internal`, or another routable host only for a non-standard topology. Loopback URLs are rewritten automatically by the driver. The bare gateway IP (`192.168.127.1`) only carries gvproxy's own services and will not reach host-bound ports. |
| `state_dir` | `target/openshell-vm-driver` | Per-sandbox overlay disks, console logs, image cache, and private `run/compute-driver.sock` UDS. |
| `grpc_endpoint` | empty | Required. URL the native host supervisor uses to reach the gateway. Host loopback such as `http://127.0.0.1:<port>` is valid. Legacy guest aliases are normalized to host loopback. This endpoint is never sent into the VM. |
| `state_dir` | `target/openshell-vm-driver` | Per-sandbox overlay disks, console logs, image cache, and private `run/compute-driver.sock` UDS. Relative paths are resolved to absolute paths at driver startup. |
| `driver_dir` | unset | Override the directory searched for `openshell-driver-vm`. |
| `default_image` | OpenShell base image | Sandbox image used when a create request omits one. |
| `bootstrap_image` | unset | VM runtime image used as the immutable bootstrap root disk. Defaults to the sandbox image when unset. |
@@ -152,8 +158,10 @@ Select the VM driver with `--compute-driver vm`, `OPENSHELL_COMPUTE_DRIVER=vm`,
| `mem_mib` | `2048` | Memory per sandbox, in MiB. |
| `overlay_disk_mib` | `4096` | Sparse writable overlay disk size per sandbox, in MiB. |
| `krun_log_level` | `1` | libkrun verbosity (0-5). |
| `sandbox_uid` / `sandbox_gid` | image `sandbox` account, otherwise `1000` / UID | Explicit values override the image account; when both are omitted, a supplied image `sandbox` account is preserved and an image without one gets `1000:1000`. Each overlay records its effective UID/GID. During migration, an unmarked overlay recovers identity from its upper layer or prepared rootfs, an explicit override, or the current image. Legacy `10001:10001` is retained only when persisted state reports it. |
| `https_proxy` | unset | Corporate forward proxy (`http://host:port` or `https://host:port`) the in-guest supervisor chains policy-approved TLS CONNECT egress through. On the libkrun backend a proxy on the gateway host's loopback must be addressed as `http://host.openshell.internal:<port>` — guest egress leaves through gvproxy, which NATs `192.168.127.254` to the host's `127.0.0.1`. The QEMU/TAP backend has no such NAT, so a gateway-host proxy URL is rejected before GPU sandbox launch; use an address routable from the guest's masqueraded egress. |
| `guest_tls_ca` | unset | Historical key name for the host supervisor's gateway CA certificate. Required when `grpc_endpoint` uses `https://`; never copied into the guest. |
| `guest_tls_cert` | unset | Historical key name for the host supervisor's client certificate; never copied into the guest. |
| `guest_tls_key` | unset | Historical key name for the host supervisor's client private key; never copied into the guest. |
| `https_proxy` | unset | Corporate forward proxy (`http://host:port` or `https://host:port`) that host control chains policy-approved TLS CONNECT egress through. Host-loopback proxy URLs work because control runs on the gateway host. |
| `no_proxy` | unset | Comma-separated bypass list for the corporate proxy only. OpenShell policy evaluation still applies. |
| `proxy_auth_file` | unset | Gateway-host path to a validated `user:pass` credential file. Staged root-only into the per-sandbox overlay and removed with the sandbox; credentials never enter logs or process arguments. |
| `proxy_auth_allow_insecure` | unset | Required with `proxy_auth_file` against an `http://` proxy: acknowledges that Basic auth is cleartext on the connection to the proxy. |
@@ -161,18 +169,7 @@ Select the VM driver with `--compute-driver vm`, `OPENSHELL_COMPUTE_DRIVER=vm`,
| `proxy_ca_bundle` | unset | Gateway-host PEM CA bundle trusted for the corporate proxy and TLS-intercepted server certificates. The driver validates it and stages it at a fixed non-secret guest path in the protected overlay. Requires `https_proxy`. |
| `provider_spiffe_workload_api_tcp_endpoint` | unset | Explicit guest-reachable `tcp:IP:port` SPIFFE Workload API listener for provider token exchange. It requires `provider_spiffe_allow_guest_tcp = true`; a host UNIX socket is never silently exposed to a VM guest. |
The proxy settings are operator-owned and deployment-level: they are not accepted through `template.driver_config.vm`, and they reach the supervisor through a protected per-sandbox argument file the driver writes into the overlay upperdir on every launch, so a sandbox image cannot forge or shadow them. Every present-but-invalid value is fatal at gateway or sandbox startup rather than degrading to a direct dial.
For gateway-managed VM drivers, configure `guest_tls_ca`, `guest_tls_cert`, and
`guest_tls_key` together under `[openshell.gateway]`; the gateway validates and
injects that bundle into only the selected local driver. The standalone
`openshell-driver-vm` CLI retains its `--guest-tls-*` inputs for independent
operation. Standalone invocations use `--grpc-endpoint` and the
`--upstream-proxy`, `--upstream-no-proxy`, `--upstream-proxy-auth-file`,
`--upstream-proxy-auth-allow-insecure`, `--upstream-proxy-connect-by-hostname`,
and `--upstream-proxy-ca-bundle` flags. Replace the removed
`--openshell-endpoint`, `--https-proxy`, `--no-proxy`, and `--proxy-*` spellings
in existing scripts.
The proxy settings are operator-owned and deployment-level: they are not accepted through `template.driver_config.vm`, and the driver passes them only to native host control. Every present-but-invalid value is fatal at gateway or sandbox startup rather than degrading to a direct dial.
See [`openshell-gateway --help`](../openshell-server/src/cli.rs) for the gateway process flag surface.
@@ -230,9 +227,17 @@ marked sandboxes without launching compute. Start removes the marker and uses
the normal persisted restore path with the existing overlay. Delete removes the
entire sandbox state directory, including a stop marker and overlay.
The driver records a terminal tombstone when the canonical main process exits.
Driver startup reports that sandbox as terminal instead of relaunching the VM,
even when the process exited successfully.
The host control writes and syncs a terminal tombstone when the canonical main
process exits, before it reports completion and while it retains the boundary
for exec and forwarding. Driver startup reports that sandbox as terminal
instead of relaunching the VM, even when the process exited successfully.
The driver embeds a platform-native host supervisor and extracts it into
`<state-dir>/host-runtime`. It accepts a cached binary only when its SHA-256
content matches the embedded supervisor and it remains an executable regular
file. Replacement is written and synced under a temporary name, then atomically
renamed into place. `OPENSHELL_VM_SUPERVISOR_BIN` remains an explicit
development override.
## Logs and debugging
@@ -244,34 +249,23 @@ RUST_LOG=openshell_server=debug,openshell_driver_vm=debug \
```
The VM guest's serial console is appended to `<state-dir>/<sandbox-id>/console.log`. Sandbox IDs must match `[A-Za-z0-9._-]{1,128}` before the driver uses them in host paths. The gateway-owned compute-driver socket lives at `<state-dir>/run/compute-driver.sock`; OpenShell creates `run/` with owner-only permissions and removes same-owner stale sockets. On clean shutdown, the gateway sends the managed driver `SIGTERM`, waits up to five seconds for it to flush telemetry and exit, then force-kills it if necessary and removes the socket. UDS clients must match the driver UID and provide the expected gateway process PID by default. Standalone same-UID UDS mode requires the explicit `--allow-same-uid-peer` development flag. TCP mode is disabled by default because it is unauthenticated; use `--allow-unauthenticated-tcp --bind-address 127.0.0.1:50061` only for local development.
The VM serial console is appended to `<state-dir>/sandboxes/<sandbox-id>/rootfs-console.log`. Host-supervisor stdout and stderr are written beside it as `supervisor.log` and `supervisor.err.log`. Sandbox IDs must match `[A-Za-z0-9._-]{1,128}` before the driver uses them in host paths. The gateway-owned compute-driver socket lives at `<state-dir>/run/compute-driver.sock`; OpenShell creates `run/` with owner-only permissions, removes same-owner stale sockets, and the gateway removes the socket on clean shutdown via `ManagedDriverProcess::drop`. UDS clients must match the driver UID and provide the expected gateway process PID by default. Standalone same-UID UDS mode requires the explicit `--allow-same-uid-peer` development flag. TCP mode is disabled by default because it is unauthenticated; use `--allow-unauthenticated-tcp --bind-address 127.0.0.1:50061` only for local development.
## Host-side nftables rules
## Network isolation
The VM driver creates a per-VM nftables table on the host (`openshell_vm_vmtap_<id>`) with three chains. These rules serve two purposes: NAT infrastructure (required for VM connectivity) and defense-in-depth host isolation. Primary security enforcement — proxy-only egress and bypass detection — is handled by the sandbox supervisor's own nftables rules inside the VM guest.
**`postrouting` (NAT):** Masquerades outbound VM traffic so it can be routed from the VM's private subnet to the external network. This chain handles forwarded traffic (VM → internet), not traffic destined for the host.
**`forward` (defense-in-depth):** Accepts all outbound traffic from the VM (security enforcement happens guest-side) and accepts established/related response traffic back to the VM. Drops unsolicited inbound connections to the VM from the broader network. This chain handles forwarded traffic only — packets transiting the host between the TAP interface and other interfaces.
**`input` (defense-in-depth):** Accepts traffic from the VM to the gateway port on the host. Drops all other traffic from the VM destined for the host itself. This limits what a compromised guest can reach on the host to the gateway service only.
The `input` and `postrouting` chains handle different traffic paths: `input` covers packets addressed to the host (VM → host), while `postrouting` covers packets the host is forwarding on behalf of the VM (VM → internet). A packet from the VM goes through one path or the other, never both.
All chains use `policy accept`, so non-TAP traffic is unaffected. Because nftables evaluates multiple base chains on the same hook independently, host firewalls interact with these rules as follows:
- **Open host (no other firewall):** Our chains are the only filter. The defense-in-depth drop rules block unsolicited inbound and non-gateway host access. Non-TAP traffic passes through.
- **Restrictive host firewall (e.g. firewalld):** The host firewall's chains may additionally drop TAP traffic that our chains accept. A `drop` verdict from any chain is final — our `accept` cannot override it. If VM connectivity fails, verify that the host firewall allows forwarding and input for `vmtap-*` interfaces.
Each table is created atomically via `nft -f` on VM start and torn down atomically via `nft delete table` when the VM is destroyed.
VM sandboxes boot without a virtual NIC. The guest exposes only the protected
vsock channel used by `openshell-sandbox`; `openshell-supervisor` performs DNS,
policy evaluation, and external networking on the host. The driver does not
create TAP devices or install nftables/iptables rules.
## Prerequisites
- macOS on Apple Silicon, or Linux on aarch64/x86_64 with KVM
- Rust toolchain
- e2fsprogs (`mke2fs` or `mkfs.ext4`, plus `debugfs`) for root and overlay disk image creation, identity inspection, and QEMU environment injection. Explicit `sandbox_uid`/`sandbox_gid` values do not remove this runtime prerequisite.
- Guest-supervisor cross-compile toolchain (needed on macOS, and on Linux when host arch ≠ guest arch):
- Matching rustup target: `rustup target add aarch64-unknown-linux-gnu` (or `x86_64-unknown-linux-gnu` for an amd64 guest)
- `cargo install --locked cargo-zigbuild` and `brew install zig` (or distro equivalent). `vm:supervisor` uses `cargo zigbuild` to cross-compile the in-VM `openshell-sandbox` supervisor binary.
- e2fsprogs (`mke2fs` or `mkfs.ext4`, plus `debugfs`) for root and overlay disk image creation and QEMU environment injection
- Guest-sandbox cross-compile toolchain (needed on macOS, and on Linux when host arch differs from the guest):
- Matching static guest target: `rustup target add aarch64-unknown-linux-musl` (or `x86_64-unknown-linux-musl` for an amd64 guest)
- `cargo install --locked cargo-zigbuild` and `brew install zig` (or distro equivalent). `vm:supervisor` cross-compiles the Linux guest `openshell-sandbox` and its matching `openshell-supervisor`.
- [mise](https://mise.jdx.dev/) task runner
- Docker or Podman socket on the local CLI/gateway host when building an image
before `openshell sandbox create --from <image>`; the VM driver exports the
@@ -302,11 +296,11 @@ The RPM gateway package is configured for the Podman driver.
On Apple Silicon macOS, `install.sh` stages the generated `openshell.rb`
formula from the selected release in the `nvidia/openshell` Homebrew tap.
Homebrew installs `openshell`, `openshell-gateway`, and
`openshell-driver-vm`, ad-hoc signs the driver with the Hypervisor entitlement
in `post_install`, and owns the `brew services` gateway lifecycle. The service
also leaves `OPENSHELL_COMPUTE_DRIVER` unset so driver choice remains automatic unless
the user explicitly overrides it.
Homebrew installs `openshell`, `openshell-gateway`, and the self-contained
`openshell-driver-vm` with its embedded native supervisor. It ad-hoc signs the
driver with the Hypervisor entitlement in `post_install` and owns the `brew
services` gateway lifecycle. The service also leaves `OPENSHELL_DRIVERS` unset
so driver choice remains automatic unless the user explicitly overrides it.
## TODOs
crates/openshell-driver-vm/build.rs
+25 -5
View File
@@ -3,13 +3,17 @@
//! Build script for openshell-driver-vm.
//!
//! This crate embeds the sandbox supervisor plus the minimal libkrun runtime
//! This crate embeds the sandbox, host supervisor, and minimal libkrun runtime
//! artifacts it needs to boot VMs without a separate VM runtime binary.
use std::path::{Path, PathBuf};
use std::{env, fs};
fn main() {
if env::var_os("CARGO_FEATURE_COMPUTE_DRIVER").is_none() {
return;
}
println!("cargo:rerun-if-env-changed=OPENSHELL_VM_RUNTIME_COMPRESSED_DIR");
if let Ok(dir) = env::var("OPENSHELL_VM_RUNTIME_COMPRESSED_DIR") {
@@ -19,8 +23,9 @@ fn main() {
"libkrunfw.so.5.zst",
"libkrun.dylib.zst",
"libkrunfw.5.dylib.zst",
"gvproxy.zst",
"openshell-sandbox.zst",
"openshell-supervisor.zst",
"openshell-vm-init.zst",
"umoci.zst",
] {
println!("cargo:rerun-if-changed={dir}/{name}");
@@ -38,7 +43,14 @@ fn main() {
println!("cargo:warning=VM runtime not available for {target_os}-{target_arch}");
generate_stub_resources(
&out_dir,
&["libkrun", "libkrunfw", "openshell-sandbox.zst", "umoci.zst"],
&[
"libkrun",
"libkrunfw",
"openshell-sandbox.zst",
"openshell-supervisor.zst",
"openshell-vm-init.zst",
"umoci.zst",
],
);
return;
}
@@ -54,8 +66,9 @@ fn main() {
&[
&format!("{libkrun_name}.zst"),
&format!("{libkrunfw_name}.zst"),
"gvproxy.zst",
"openshell-sandbox.zst",
"openshell-supervisor.zst",
"openshell-vm-init.zst",
"umoci.zst",
],
);
@@ -74,11 +87,18 @@ fn main() {
format!("{libkrunfw_name}.zst"),
format!("{libkrunfw_name}.zst"),
),
("gvproxy.zst".to_string(), "gvproxy.zst".to_string()),
(
"openshell-sandbox.zst".to_string(),
"openshell-sandbox.zst".to_string(),
),
(
"openshell-supervisor.zst".to_string(),
"openshell-supervisor.zst".to_string(),
),
(
"openshell-vm-init.zst".to_string(),
"openshell-vm-init.zst".to_string(),
),
("umoci.zst".to_string(), "umoci.zst".to_string()),
];
crates/openshell-driver-vm/runtime/README.md
+14 -11
View File
@@ -11,15 +11,18 @@ runtime/
openshell.kconfig
```
`openshell-driver-vm` embeds libkrun, libkrunfw, gvproxy, umoci for guest-side
OCI image unpacking, and the bundled `openshell-sandbox` supervisor.
`openshell-driver-vm` embeds libkrun, libkrunfw, umoci for guest-side OCI image
unpacking, and the portable capability-free `openshell-sandbox` role.
VMs do not attach a guest NIC. The boundary carries control, mediated network,
and DNS streams over the authenticated vsock channel.
## Why
The stock `libkrunfw` kernel does not include the bridge, netfilter,
conntrack, cgroup, seccomp, and Landlock features the sandbox supervisor needs
inside each microVM. `kernel/openshell.kconfig` extends the libkrunfw kernel so
VM sandboxes can run the same supervisor enforcement path as other backends.
The stock `libkrunfw` kernel does not include every cgroup, seccomp, and
Landlock feature the sandbox needs inside each microVM.
`kernel/openshell.kconfig` extends the libkrunfw kernel so VM sandboxes retain
guest-local process, network-syscall, and filesystem enforcement while the
supervisor runs on the host.
## Build Scripts
@@ -27,7 +30,7 @@ VM sandboxes can run the same supervisor enforcement path as other backends.
|---|---|---|
| `tasks/scripts/vm/build-libkrun.sh` | Linux | Builds libkrunfw and libkrun from source with the custom kernel config |
| `tasks/scripts/vm/build-libkrun-macos.sh` | macOS | Builds portable libkrunfw and libkrun from a prebuilt `kernel.c` |
| `tasks/scripts/vm/package-vm-runtime.sh` | Any | Packages `vm-runtime-<platform>.tar.zst` with libraries, gvproxy, umoci, and provenance |
| `tasks/scripts/vm/package-vm-runtime.sh` | Any | Packages `vm-runtime-<platform>.tar.zst` with libraries, umoci, and provenance |
| `tasks/scripts/vm/download-kernel-runtime.sh` | Any | Downloads runtime tarballs from the `vm-runtime` release and stages compressed files |
## Local Flow
@@ -36,12 +39,12 @@ VM sandboxes can run the same supervisor enforcement path as other backends.
# Download the current pre-built runtime and stage compressed artifacts
mise run vm:setup
# Build the bundled guest supervisor
# Build the portable Linux guest sandbox and static guest-init helper
mise run vm:supervisor
# Build the gateway and VM driver with embedded runtime artifacts
# Build the gateway, native host supervisor, and VM driver
OPENSHELL_VM_RUNTIME_COMPRESSED_DIR=$PWD/target/vm-runtime-compressed \
cargo build -p openshell-gateway -p openshell-driver-vm
cargo build -p openshell-gateway -p openshell-supervisor -p openshell-driver-vm
```
Use `FROM_SOURCE=1 mise run vm:setup` to build the runtime from source instead
@@ -62,7 +65,7 @@ publish the driver binary next to `openshell-gateway`.
## Provenance
`package-vm-runtime.sh` writes `provenance.json` into each runtime tarball with
the platform, libkrunfw commit, kernel version, gvproxy and umoci versions,
the platform, libkrunfw commit, kernel version, and umoci version,
GitHub SHA, and build time. The driver logs this metadata when it extracts and
loads a runtime bundle.
crates/openshell-driver-vm/runtime/kernel/openshell.kconfig
+11 -110
View File
@@ -1,136 +1,37 @@
# Custom kernel config fragment for libkrunfw (OpenShell VM)
#
# This fragment is applied on top of libkrunfw's base kernel config
# to enable bridge CNI, netfilter/iptables, and conntrack support
# required for Kubernetes pod networking in the VM.
# This fragment is applied on top of libkrunfw's base kernel config. VM
# sandboxes have no virtual NIC; the base loopback, TCP/UDP, Unix-socket,
# route-netlink, and vsock support is enough for the capability-free sandbox.
#
# Apply with: scripts/merge-kconfig.sh <base_config> <this_file>
#
# See also: check-vm-capabilities.sh for runtime verification.
# ── Root disk transport and filesystem ─────────────────────────────────
# Root disk transport and filesystem.
CONFIG_BLOCK=y
CONFIG_BLK_DEV=y
CONFIG_VIRTIO_BLK=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_USE_FOR_EXT2=y
# ── Network Namespaces (required for pod isolation) ─────────────────────
CONFIG_NET_NS=y
CONFIG_NAMESPACES=y
# ── Virtual Ethernet (veth pairs for pod networking) ────────────────────
CONFIG_VETH=y
# ── Linux Bridge (required for bridge CNI plugin) ──────────────────────
CONFIG_BRIDGE=y
CONFIG_BRIDGE_NETFILTER=y
CONFIG_BRIDGE_IGMP_SNOOPING=y
# ── Netfilter framework ────────────────────────────────────────────────
CONFIG_NETFILTER=y
CONFIG_NETFILTER_ADVANCED=y
CONFIG_NETFILTER_INGRESS=y
CONFIG_NETFILTER_NETLINK=y
CONFIG_NETFILTER_NETLINK_QUEUE=y
CONFIG_NETFILTER_NETLINK_LOG=y
# ── Connection tracking (required for NAT and kube-proxy) ──────────────
CONFIG_NF_CONNTRACK=y
CONFIG_NF_CT_NETLINK=y
CONFIG_NF_CONNTRACK_EVENTS=y
CONFIG_NF_CONNTRACK_TIMEOUT=y
CONFIG_NF_CONNTRACK_TIMESTAMP=y
# ── Netfilter xtables match modules (required by kube-proxy & kubelet) ─
# kube-proxy uses xt_conntrack for stateful rules and xt_comment for
# labeling chains. Without these, iptables fails with:
# "Couldn't load match 'conntrack': No such file or directory"
CONFIG_NETFILTER_XTABLES=y
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
CONFIG_NETFILTER_XT_MATCH_COMMENT=y
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y
CONFIG_NETFILTER_XT_MATCH_MARK=y
CONFIG_NETFILTER_XT_MATCH_STATISTIC=y
CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y
CONFIG_NETFILTER_XT_MATCH_RECENT=y
CONFIG_NETFILTER_XT_MATCH_LIMIT=y
CONFIG_NETFILTER_XT_TARGET_LOG=y
CONFIG_NETFILTER_XT_TARGET_MARK=y
CONFIG_NETFILTER_XT_TARGET_CONNMARK=y
CONFIG_NETFILTER_XT_MATCH_CONNMARK=y
# ── NAT (required for service VIP / DNAT / SNAT) ──────────────────────
CONFIG_NF_NAT=y
CONFIG_NF_NAT_MASQUERADE_IPV4=y
# ── iptables (CNI bridge masquerade + compat) ──────────────────────────
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_NAT=y
CONFIG_IP_NF_MANGLE=y
CONFIG_IP_NF_TARGET_MASQUERADE=y
CONFIG_IP_NF_TARGET_REJECT=y
# ── nftables (kube-proxy nftables mode — primary proxy backend) ─────────
# kube-proxy nftables proxier requires: numgen (random LB), fib (local
# address detection), counter, ct, nat, masq, reject, limit, redir.
CONFIG_NF_TABLES=y
CONFIG_NF_TABLES_INET=y
CONFIG_NFT_CT=y
CONFIG_NFT_NAT=y
CONFIG_NFT_MASQ=y
CONFIG_NFT_REJECT=y
CONFIG_NFT_COMPAT=y
CONFIG_NFT_NUMGEN=y
CONFIG_NFT_FIB_IPV4=y
CONFIG_NFT_FIB_IPV6=y
CONFIG_NFT_LIMIT=y
CONFIG_NFT_LOG=y
CONFIG_NFT_REDIR=y
CONFIG_NFT_TPROXY=y
# ── IP forwarding and routing (required for pod-to-pod) ────────────────
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_NET_IP_TUNNEL=y
# ── IPVS (optional: kube-proxy IPVS mode) ─────────────────────────────
CONFIG_IP_VS=y
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_RR=y
CONFIG_IP_VS_WRR=y
CONFIG_IP_VS_SH=y
CONFIG_IP_VS_NFCT=y
# ── Misc networking required by Kubernetes ─────────────────────────────
CONFIG_NET_SCH_HTB=y
CONFIG_NET_CLS_CGROUP=y
CONFIG_CGROUP_NET_PRIO=y
CONFIG_CGROUP_NET_CLASSID=y
# ── Dummy interface (fallback networking) ──────────────────────────────
CONFIG_DUMMY=y
# ── TUN/TAP (used by some CNI plugins) ────────────────────────────────
CONFIG_TUN=y
# ── Cgroups (already in base, ensure v2 is available) ──────────────────
# Cgroups used for process supervision and resource limits.
CONFIG_CGROUPS=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_PIDS=y
CONFIG_MEMCG=y
# ── Disable kernel headers archive (avoids cpio issues in CI) ──────────
# Disable the kernel headers archive to avoid cpio issues in CI.
# CONFIG_IKHEADERS is not set
# ── POSIX message queues (required by runc to mount /dev/mqueue in containers) ─
# POSIX message queues used by OCI workloads.
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
# ── Security features required by the sandbox runtime ───────────────────
# Capability-free sandbox enforcement.
CONFIG_SECURITY=y
CONFIG_SECURITY_LANDLOCK=y
CONFIG_LSM="landlock,lockdown,yama,loadpin,safesetid,ipe,bpf"
CONFIG_SECCOMP=y
CONFIG_SECCOMP_FILTER=y
crates/openshell-driver-vm/runtime/pins.env
-4
View File
@@ -29,10 +29,6 @@ COMMUNITY_SANDBOX_IMAGE="${COMMUNITY_SANDBOX_IMAGE:-ghcr.io/nvidia/openshell-com
# during local development. Override via IMAGE_REPO_BASE and IMAGE_TAG
# environment variables (defaults: openshell/gateway:dev).
# ── gvproxy (networking proxy) ──────────────────────────────────────────
# Repo: https://github.com/containers/gvisor-tap-vsock
GVPROXY_VERSION="${GVPROXY_VERSION:-v0.8.9}"
# ── umoci (guest OCI unpacker) ──────────────────────────────────────────
# Repo: https://github.com/opencontainers/umoci
UMOCI_VERSION="${UMOCI_VERSION:-v0.6.0}"
crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh
+53 -372
View File
@@ -3,9 +3,9 @@
# SPDX-License-Identifier: Apache-2.0
# Minimal init for sandbox VMs. Runs as PID 1 inside the guest, mounts the
# essential filesystems, configures networking (gvproxy DHCP or TAP static),
# optionally loads NVIDIA GPU drivers, then execs the OpenShell sandbox
# supervisor.
# essential filesystems, optionally loads NVIDIA GPU drivers, then execs the
# portable VM sandbox. Workload networking crosses the authenticated
# boundary channel; the VM does not receive a network interface.
set -euo pipefail
@@ -14,26 +14,9 @@ set -euo pipefail
unset KRUN_INIT_PID1
BOOT_START=$(date +%s%3N 2>/dev/null || date +%s)
# gvisor-tap-vsock subnet layout:
# 192.168.127.1 — gateway: gvproxy's DNS / DHCP / HTTP API. Does NOT
# proxy arbitrary host ports.
# 192.168.127.254 — host-loopback: NAT-rewritten to host's 127.0.0.1 by
# gvproxy's TCP/UDP/ICMP forwarder. Use this address
# (or any of the host.* hostnames below) to reach a
# service the host is listening on.
# The host.openshell.internal / host.containers.internal /
# host.docker.internal DNS records served by gvproxy's embedded resolver
# point at 192.168.127.254. We mirror that in /etc/hosts so the supervisor
# can reach the gateway even when gvproxy's DNS is not in resolv.conf
# (e.g. DHCP failed and we fell back to 8.8.8.8).
GVPROXY_GATEWAY_IP="192.168.127.1"
GVPROXY_HOST_LOOPBACK_IP="192.168.127.254"
GATEWAY_IP="$GVPROXY_GATEWAY_IP"
SANDBOX_OWNER_NORMALIZED_MARKER="/opt/openshell/.sandbox-owner-normalized"
GPU_ENABLED="${GPU_ENABLED:-false}"
VM_NET_IP="${VM_NET_IP:-}"
VM_NET_GW="${VM_NET_GW:-}"
VM_NET_DNS="${VM_NET_DNS:-}"
ts() {
local now
@@ -133,6 +116,10 @@ ensure_target_runtime() {
cp /opt/openshell/bin/openshell-sandbox "$image_root/opt/openshell/bin/openshell-sandbox"
chmod 0755 "$image_root/opt/openshell/bin/openshell-sandbox"
fi
if [ -x /opt/openshell/bin/openshell-vm-init ]; then
cp /opt/openshell/bin/openshell-vm-init "$image_root/opt/openshell/bin/openshell-vm-init"
chmod 0755 "$image_root/opt/openshell/bin/openshell-vm-init"
fi
touch "$image_root/etc/passwd" "$image_root/etc/group" "$image_root/etc/shadow" "$image_root/etc/gshadow"
if [ "$replace_account" -eq 1 ]; then
@@ -208,67 +195,6 @@ prepare_guest_image_rootfs() {
rm -rf "$payload_dir"
}
# Driver-owned arguments appended to the supervisor's command line.
#
# The VM driver cannot build the supervisor's argv the way the container
# drivers do, so it writes the arguments it chose into the overlay upperdir
# and this script appends them verbatim. Populated by
# read_supervisor_extra_args; empty until then.
SUPERVISOR_EXTRA_ARGS=()
# Upper bound on driver-supplied supervisor arguments.
#
# The corporate proxy settings are the only producer today and top out at ten
# entries. The cap exists so a corrupt or oversized file cannot expand into an
# unbounded command line.
SUPERVISOR_EXTRA_ARGS_MAX=32
read_supervisor_extra_args() {
# Read the driver-authored supervisor argument list, one argument per
# line, verbatim -- no word splitting, globbing, or expansion, so values
# containing spaces (e.g. a NO_PROXY list) survive intact.
#
# Security: this is the operator-owned egress boundary. The driver writes
# this file into the overlay upperdir on every launch, including an empty
# file when it has no arguments to pass, so the upperdir copy always
# shadows the read-only image layer. A sandbox image can therefore neither
# supply its own supervisor arguments by baking a file at this path nor
# disable the operator's by omitting one. A missing file means the driver
# passed nothing; a file it cannot read means the overlay is broken, and
# we fail closed rather than start a supervisor with a silently truncated
# egress configuration.
local args_file
args_file="$(root_path /opt/openshell/supervisor-args)"
SUPERVISOR_EXTRA_ARGS=()
if [ ! -f "$args_file" ]; then
return 0
fi
if [ ! -r "$args_file" ]; then
ts "FATAL: supervisor argument list ${args_file} is not readable"
exit 1
fi
local arg
while IFS= read -r arg; do
# render_guest_supervisor_args never emits a blank line, so one means
# the file was truncated or tampered with after the driver wrote it.
if [ -z "$arg" ]; then
ts "FATAL: empty entry in supervisor argument list"
exit 1
fi
if [ "${#SUPERVISOR_EXTRA_ARGS[@]}" -ge "$SUPERVISOR_EXTRA_ARGS_MAX" ]; then
ts "FATAL: supervisor argument list exceeds ${SUPERVISOR_EXTRA_ARGS_MAX} entries"
exit 1
fi
SUPERVISOR_EXTRA_ARGS+=("$arg")
done < "$args_file"
if [ "${#SUPERVISOR_EXTRA_ARGS[@]}" -gt 0 ]; then
ts "supervisor arguments from driver: ${#SUPERVISOR_EXTRA_ARGS[@]} entries"
fi
}
exec_supervisor_in_newroot() {
local chroot_bin
local bootstrap="/.openshell-bootstrap"
@@ -369,170 +295,6 @@ setup_overlay_root() {
run_post_overlay_setup
}
parse_endpoint() {
local endpoint="$1"
local scheme rest authority path host port
case "$endpoint" in
*://*)
scheme="${endpoint%%://*}"
rest="${endpoint#*://}"
;;
*)
return 1
;;
esac
authority="${rest%%/*}"
path="${rest#"$authority"}"
if [ "$path" = "$rest" ]; then
path=""
fi
if [[ "$authority" =~ ^\[([^]]+)\]:(.+)$ ]]; then
host="${BASH_REMATCH[1]}"
port="${BASH_REMATCH[2]}"
elif [[ "$authority" =~ ^\[([^]]+)\]$ ]]; then
host="${BASH_REMATCH[1]}"
port=""
elif [[ "$authority" == *:* ]]; then
host="${authority%%:*}"
port="${authority##*:}"
else
host="$authority"
port=""
fi
if [ -z "$port" ]; then
case "$scheme" in
https) port="443" ;;
*) port="80" ;;
esac
fi
printf '%s\n%s\n%s\n%s\n' "$scheme" "$host" "$port" "$path"
}
tcp_probe() {
local host="$1"
local port="$2"
if command -v timeout >/dev/null 2>&1; then
timeout 2 bash -c "exec 3<>/dev/tcp/\$1/\$2" _ "$host" "$port" >/dev/null 2>&1
else
bash -c "exec 3<>/dev/tcp/\$1/\$2" _ "$host" "$port" >/dev/null 2>&1
fi
}
ensure_host_gateway_aliases() {
# Seed /etc/hosts with the well-known gvproxy hostnames so the supervisor
# can reach the OpenShell server even when gvproxy's built-in DNS is not
# in resolv.conf (e.g. when DHCP fails and we fall back to 8.8.8.8).
#
# Critical distinction: host.* aliases point at the gvproxy *host-loopback*
# IP (192.168.127.254), not the gateway IP (192.168.127.1). Only the
# host-loopback IP carries NAT rewriting to the host's 127.0.0.1 — the
# gateway IP only listens on gvproxy's own service ports (DNS:53, DHCP,
# HTTP API:80). Pinning host.containers.internal to the gateway IP
# silently breaks guest→host port reachability for arbitrary ports.
local host_aliases="host.openshell.internal host.containers.internal host.docker.internal"
local gateway_aliases="gateway.containers.internal"
local filter='(^|[[:space:]])(host\.openshell\.internal|host\.containers\.internal|host\.docker\.internal|gateway\.containers\.internal)([[:space:]]|$)'
write_host_gateway_aliases "$(root_path /etc/hosts)" "$(root_path "/tmp/openshell-hosts.$$.tmp")" || true
if [ -n "${ROOT_PREFIX:-}" ]; then
write_host_gateway_aliases "/etc/hosts" "/tmp/openshell-hosts.$$.tmp" || true
fi
}
write_host_gateway_aliases() {
local hosts_path="$1"
local hosts_tmp="$2"
mkdir -p "$(dirname "$hosts_path")" 2>/dev/null || true
mkdir -p "$(dirname "$hosts_tmp")" 2>/dev/null || true
if [ -f "$hosts_path" ]; then
grep -vE "$filter" "$hosts_path" > "$hosts_tmp" || true
else
: > "$hosts_tmp"
fi
# In TAP/GPU mode, GATEWAY_IP is overridden to VM_NET_GW (the host-side
# of the TAP), and the gateway is reachable directly there. In gvproxy
# mode, host.openshell.internal etc. need GVPROXY_HOST_LOOPBACK_IP
# (192.168.127.254) which is gvproxy's host-NAT entry, while
# gateway.containers.internal points at the gvproxy gateway itself.
if [ "${GATEWAY_IP}" = "${GVPROXY_GATEWAY_IP}" ]; then
printf '%s %s\n' "$GVPROXY_HOST_LOOPBACK_IP" "$host_aliases" >> "$hosts_tmp"
printf '%s %s\n' "$GVPROXY_GATEWAY_IP" "$gateway_aliases" >> "$hosts_tmp"
else
# TAP networking: gateway and host are both reachable at GATEWAY_IP.
printf '%s %s %s\n' "$GATEWAY_IP" "$host_aliases" "$gateway_aliases" >> "$hosts_tmp"
fi
if ! cat "$hosts_tmp" > "$hosts_path" 2>/dev/null; then
rm -f "$hosts_tmp"
ts "WARNING: could not update ${hosts_path}"
return 1
fi
rm -f "$hosts_tmp"
}
rewrite_openshell_endpoint_if_needed() {
local endpoint="${OPENSHELL_ENDPOINT:-}"
[ -n "$endpoint" ] || return 0
local parsed
if ! parsed="$(parse_endpoint "$endpoint")"; then
ts "WARNING: could not parse OPENSHELL_ENDPOINT=$endpoint"
return 0
fi
local scheme host port path
scheme="$(printf '%s\n' "$parsed" | sed -n '1p')"
host="$(printf '%s\n' "$parsed" | sed -n '2p')"
port="$(printf '%s\n' "$parsed" | sed -n '3p')"
path="$(printf '%s\n' "$parsed" | sed -n '4p')"
if tcp_probe "$host" "$port"; then
return 0
fi
# Probe candidates in preference order. Hostnames first for informative
# log output, then a bare IP as a final safety net. In gvproxy mode the
# bare IP is the host-loopback (192.168.127.254). In TAP/GPU mode it's
# the TAP host gateway.
local fallback_ip="$GVPROXY_HOST_LOOPBACK_IP"
if [ "${GATEWAY_IP}" != "${GVPROXY_GATEWAY_IP}" ]; then
fallback_ip="$GATEWAY_IP"
fi
local candidates="host.openshell.internal host.containers.internal host.docker.internal"
if [ "$scheme" != "https" ]; then
candidates="${candidates} ${fallback_ip}"
fi
for candidate in $candidates; do
if [ "$candidate" = "$host" ]; then
continue
fi
if tcp_probe "$candidate" "$port"; then
local authority="$candidate"
if ! { [ "$scheme" = "http" ] && [ "$port" = "80" ]; } \
&& ! { [ "$scheme" = "https" ] && [ "$port" = "443" ]; }; then
authority="${authority}:${port}"
fi
export OPENSHELL_ENDPOINT="${scheme}://${authority}${path}"
ts "rewrote OPENSHELL_ENDPOINT to ${OPENSHELL_ENDPOINT}"
return 0
fi
done
if [ "$scheme" = "https" ]; then
ts "WARNING: could not preflight HTTPS OpenShell endpoint ${host}:${port}; preserving hostname for TLS verification"
return 0
fi
ts "WARNING: could not reach OpenShell endpoint ${host}:${port}"
}
create_gpu_device_nodes_mknod() {
# Mode 666 is intentional: single-tenant microVM with the VM itself as the
# isolation boundary. The sandbox user is the only non-root user.
@@ -801,117 +563,28 @@ run_post_overlay_setup() {
mount -t cgroup2 cgroup2 "$(root_path /sys/fs/cgroup)" 2>/dev/null &
wait
# Allow nftables LOG rules to work in non-init network namespaces.
# Without this, the kernel's nf_log_syslog silently suppresses output
# from the sandbox's network namespace.
if [ -f /proc/sys/net/netfilter/nf_log_all_netns ]; then
echo 1 > /proc/sys/net/netfilter/nf_log_all_netns 2>/dev/null || true
fi
reconcile_sandbox_account
setup_sandbox_workdir
configure_hostname
ip link set lo up 2>/dev/null || true
# Networking: use TAP static config if VM_NET_IP is set (QEMU path),
# otherwise fall back to gvproxy DHCP on eth0 (libkrun path).
if [ -n "${VM_NET_IP}" ] && [ -n "${VM_NET_GW}" ]; then
ts "configuring TAP networking (static ${VM_NET_IP} gw ${VM_NET_GW})"
GATEWAY_IP="${VM_NET_GW}"
TAP_NIC=""
NIC_WAIT=0
while [ -z "$TAP_NIC" ] && [ "$NIC_WAIT" -lt 10 ]; do
for candidate in eth0 ens3 enp0s2; do
if ip link show "$candidate" >/dev/null 2>&1 && [ "$candidate" != "lo" ]; then
TAP_NIC="$candidate"
break
fi
done
if [ -z "$TAP_NIC" ]; then
for sys_nic in /sys/class/net/*; do
[ -e "$sys_nic" ] || continue
candidate="${sys_nic##*/}"
if ip link show "$candidate" >/dev/null 2>&1 && [ "$candidate" != "lo" ]; then
TAP_NIC="$candidate"
break
fi
done
fi
if [ -z "$TAP_NIC" ]; then
sleep 1
NIC_WAIT=$((NIC_WAIT + 1))
fi
done
if [ -n "$TAP_NIC" ]; then
ts "using NIC ${TAP_NIC} for TAP networking"
ip link set "$TAP_NIC" up 2>/dev/null || true
ip addr add "${VM_NET_IP}/30" dev "$TAP_NIC" 2>/dev/null || true
ip route add default via "${VM_NET_GW}" 2>/dev/null || true
else
ts "WARNING: no network interface found for TAP networking"
if ! /opt/openshell/bin/openshell-vm-init prepare-network; then
ts "FATAL: failed to bring up the loopback interface"
exit 1
fi
if [ -n "${VM_NET_DNS}" ]; then
echo "nameserver ${VM_NET_DNS}" > "$(root_path /etc/resolv.conf)"
elif [ ! -s "$(root_path /etc/resolv.conf)" ]; then
echo "nameserver 8.8.8.8" > "$(root_path /etc/resolv.conf)"
echo "nameserver 8.8.4.4" >> "$(root_path /etc/resolv.conf)"
# The capability-free sandbox owns a loopback-only DNS relay. Guest init
# grants the low port before handing control to the zero-capability UID.
if ! echo 0 > /proc/sys/net/ipv4/ip_unprivileged_port_start; then
ts "FATAL: failed to permit the unprivileged DNS relay to bind port 53"
exit 1
fi
cat >"$(root_path /etc/resolv.conf)" <<'EOF'
nameserver 127.0.0.53
options timeout:2 attempts:2
EOF
ensure_host_gateway_aliases
elif ip link show eth0 >/dev/null 2>&1; then
ts "detected eth0 (gvproxy networking)"
ip link set eth0 up 2>/dev/null || true
if command -v udhcpc >/dev/null 2>&1; then
UDHCPC_SCRIPT="$(root_path /run/openshell-udhcpc.script)"
mkdir -p "$(dirname "$UDHCPC_SCRIPT")"
cat > "$UDHCPC_SCRIPT" <<'DHCP_SCRIPT'
#!/bin/sh
case "$1" in
bound|renew)
ip addr flush dev "$interface"
ip addr add "$ip/$mask" dev "$interface"
if [ -n "$router" ]; then
ip route add default via "$router" dev "$interface"
fi
if [ -n "$dns" ]; then
resolv_conf="${OPENSHELL_RESOLV_CONF:-/etc/resolv.conf}"
mkdir -p "$(dirname "$resolv_conf")" 2>/dev/null || true
: > "$resolv_conf" 2>/dev/null || true
for d in $dns; do
echo "nameserver $d" >> "$resolv_conf" 2>/dev/null || true
done
fi
;;
esac
DHCP_SCRIPT
chmod +x "$UDHCPC_SCRIPT"
if ! OPENSHELL_RESOLV_CONF="$(root_path /etc/resolv.conf)" \
udhcpc -i eth0 -f -q -n -T 1 -t 3 -A 1 -s "$UDHCPC_SCRIPT" 2>&1; then
ts "WARNING: DHCP failed, falling back to static config"
ip addr add 192.168.127.2/24 dev eth0 2>/dev/null || true
ip route add default via "$GVPROXY_GATEWAY_IP" 2>/dev/null || true
fi
else
ts "no DHCP client, using static config"
ip addr add 192.168.127.2/24 dev eth0 2>/dev/null || true
ip route add default via "$GVPROXY_GATEWAY_IP" 2>/dev/null || true
fi
if [ ! -s "$(root_path /etc/resolv.conf)" ]; then
echo "nameserver 8.8.8.8" > "$(root_path /etc/resolv.conf)"
echo "nameserver 8.8.4.4" >> "$(root_path /etc/resolv.conf)"
fi
ensure_host_gateway_aliases
else
ts "WARNING: no network interface found; supervisor will start without guest egress"
fi
# The boundary transport mediates network and DNS requests. Only loopback is
# configured in the guest; no public resolver or guest NIC is needed.
export HOME=/sandbox
export USER=sandbox
@@ -938,34 +611,42 @@ fi
run_openshell_init_dropins
rewrite_openshell_endpoint_if_needed
# Log supervisor connectivity state for debugging stuck-in-Provisioning issues
if [ -n "${OPENSHELL_ENDPOINT:-}" ]; then
_ep_parsed="$(parse_endpoint "$OPENSHELL_ENDPOINT" 2>/dev/null || true)"
if [ -n "$_ep_parsed" ]; then
_ep_host="$(printf '%s\n' "$_ep_parsed" | sed -n '2p')"
_ep_port="$(printf '%s\n' "$_ep_parsed" | sed -n '3p')"
if tcp_probe "$_ep_host" "$_ep_port"; then
ts "gateway reachable at ${_ep_host}:${_ep_port}"
else
ts "WARNING: gateway NOT reachable at ${_ep_host}:${_ep_port} — supervisor may fail to connect"
fi
fi
ts "OPENSHELL_ENDPOINT=${OPENSHELL_ENDPOINT}"
fi
if [ -n "${OPENSHELL_SANDBOX_ID:-}" ]; then
ts "OPENSHELL_SANDBOX_ID=${OPENSHELL_SANDBOX_ID}"
fi
read_supervisor_extra_args
set -- --workdir /sandbox "${SUPERVISOR_EXTRA_ARGS[@]+"${SUPERVISOR_EXTRA_ARGS[@]}"}"
ts "starting openshell-sandbox supervisor"
if [ "${ROOT_PREFIX:-}" = "/newroot" ]; then
exec_supervisor_in_newroot "$@"
ts "starting OpenShell VM sandbox"
_sandbox_owner="$(sandbox_owner)"
_sandbox_uid="${_sandbox_owner%%:*}"
_sandbox_gid="${_sandbox_owner##*:}"
_sandbox_bootstrap_guest="${OPENSHELL_VM_SANDBOX_BOOTSTRAP:-/.openshell/state/bootstrap.json}"
_sandbox_bootstrap="$(root_path "$_sandbox_bootstrap_guest")"
_sandbox_state_dir="${_sandbox_bootstrap%/*}"
if [ ! -f "$_sandbox_bootstrap" ]; then
ts "FATAL: capability-free sandbox bootstrap is missing"
exit 1
fi
exec /opt/openshell/bin/openshell-sandbox "$@"
chown "${_sandbox_uid}:${_sandbox_gid}" "$_sandbox_state_dir"
chmod 0700 "$_sandbox_state_dir"
for _sandbox_private_file in "$_sandbox_state_dir"/*; do
[ -f "$_sandbox_private_file" ] || continue
chown "${_sandbox_uid}:${_sandbox_gid}" "$_sandbox_private_file"
chmod 0600 "$_sandbox_private_file"
done
if [ "${OPENSHELL_VM_INIT_MODE:-sandbox}" = "capability-probe" ]; then
ts "starting capability-free VM qualification as ${_sandbox_uid}:${_sandbox_gid}"
if [ "${ROOT_PREFIX:-}" = "/newroot" ]; then
exec_supervisor_in_newroot capability-probe-launch "$_sandbox_uid" "$_sandbox_gid"
fi
exec /opt/openshell/bin/openshell-sandbox \
capability-probe-launch "$_sandbox_uid" "$_sandbox_gid"
fi
if [ "${ROOT_PREFIX:-}" = "/newroot" ]; then
exec_supervisor_in_newroot \
launch-capability-free "$_sandbox_uid" "$_sandbox_gid" "$_sandbox_bootstrap_guest"
fi
exec /opt/openshell/bin/openshell-sandbox \
launch-capability-free "$_sandbox_uid" "$_sandbox_gid" "$_sandbox_bootstrap_guest"
}
if [ "${1:-}" != "--post-overlay" ]; then
crates/openshell-driver-vm/src/bin/openshell-vm-init.rs
+240
View File
@@ -0,0 +1,240 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
//! Narrow, one-shot guest preparation used by the VM compute driver.
use std::ffi::OsString;
use std::fmt;
#[cfg(target_os = "linux")]
use std::mem::size_of;
#[cfg(target_os = "linux")]
use std::os::fd::{AsRawFd as _, FromRawFd as _, OwnedFd};
use std::process::ExitCode;
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum Command {
PrepareNetwork,
Version,
}
#[cfg(target_os = "linux")]
#[derive(Clone, Copy)]
enum InterfaceFlagOperation {
Read,
Write,
}
#[derive(Debug)]
struct InitError(String);
impl fmt::Display for InitError {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter.write_str(&self.0)
}
}
fn main() -> ExitCode {
match run(std::env::args_os().skip(1)) {
Ok(()) => ExitCode::SUCCESS,
Err(error) => {
eprintln!("openshell-vm-init: {error}");
ExitCode::FAILURE
}
}
}
fn run(args: impl IntoIterator<Item = OsString>) -> Result<(), InitError> {
match parse_command(args)? {
Command::PrepareNetwork => prepare_network(),
Command::Version => {
println!("openshell-vm-init {}", env!("CARGO_PKG_VERSION"));
Ok(())
}
}
}
fn parse_command(args: impl IntoIterator<Item = OsString>) -> Result<Command, InitError> {
let mut args = args.into_iter();
let command = args
.next()
.ok_or_else(|| InitError("expected the prepare-network command".to_string()))?;
if args.next().is_some() {
return Err(InitError("command does not accept arguments".to_string()));
}
match command.to_str() {
Some("prepare-network") => Ok(Command::PrepareNetwork),
Some("--version") => Ok(Command::Version),
_ => Err(InitError(format!(
"unknown command '{}'; expected prepare-network or --version",
command.to_string_lossy()
))),
}
}
#[cfg(target_os = "linux")]
#[allow(unsafe_code)]
fn prepare_network() -> Result<(), InitError> {
// The helper is only invoked by trusted VM guest init, before it hands the
// workload to the capability-free sandbox identity.
if unsafe { libc::geteuid() } != 0 {
return Err(InitError(
"prepare-network must run as the VM guest root user".to_string(),
));
}
// SAFETY: socket returns a fresh descriptor or -1 and does not borrow any
// caller-owned memory.
let raw_fd = unsafe { libc::socket(libc::AF_INET, libc::SOCK_DGRAM | libc::SOCK_CLOEXEC, 0) };
if raw_fd < 0 {
return Err(last_os_error("open loopback control socket"));
}
// SAFETY: raw_fd was returned successfully above and ownership transfers
// exactly once to OwnedFd.
let socket = unsafe { OwnedFd::from_raw_fd(raw_fd) };
let mut request = InterfaceRequest::loopback();
ioctl_interface_flags(
socket.as_raw_fd(),
InterfaceFlagOperation::Read,
&mut request,
)
.map_err(|error| InitError(format!("read loopback flags: {error}")))?;
let up_flag = libc::c_short::try_from(libc::IFF_UP)
.map_err(|_| InitError("platform IFF_UP value does not fit in ifreq flags".to_string()))?;
let flags = request.flags();
if flags & up_flag == 0 {
request.set_flags(flags | up_flag);
ioctl_interface_flags(
socket.as_raw_fd(),
InterfaceFlagOperation::Write,
&mut request,
)
.map_err(|error| InitError(format!("enable loopback: {error}")))?;
}
request.set_flags(0);
ioctl_interface_flags(
socket.as_raw_fd(),
InterfaceFlagOperation::Read,
&mut request,
)
.map_err(|error| InitError(format!("verify loopback flags: {error}")))?;
if request.flags() & up_flag == 0 {
return Err(InitError(
"loopback remained down after successful configuration".to_string(),
));
}
Ok(())
}
#[cfg(not(target_os = "linux"))]
fn prepare_network() -> Result<(), InitError> {
Err(InitError(
"prepare-network is only supported in Linux VM guests".to_string(),
))
}
#[cfg(target_os = "linux")]
#[repr(C)]
union InterfaceRequestData {
flags: libc::c_short,
// Linux's ifreq union is 24 bytes on the supported 64-bit guest targets.
storage: [u8; 24],
}
#[cfg(target_os = "linux")]
#[repr(C)]
struct InterfaceRequest {
name: [libc::c_char; libc::IFNAMSIZ],
data: InterfaceRequestData,
}
#[cfg(target_os = "linux")]
const _: () = assert!(size_of::<InterfaceRequest>() == size_of::<libc::ifreq>());
#[cfg(target_os = "linux")]
impl InterfaceRequest {
fn loopback() -> Self {
let mut request = Self {
name: [0; libc::IFNAMSIZ],
data: InterfaceRequestData { storage: [0; 24] },
};
request.name[0] = 108;
request.name[1] = 111;
request
}
#[allow(unsafe_code)]
fn flags(&self) -> libc::c_short {
// SAFETY: the request was most recently populated by SIOCGIFFLAGS or
// set_flags, both of which initialize the flags union member.
unsafe { self.data.flags }
}
fn set_flags(&mut self, flags: libc::c_short) {
self.data.flags = flags;
}
}
#[cfg(target_os = "linux")]
#[allow(unsafe_code)]
fn ioctl_interface_flags(
fd: std::os::fd::RawFd,
operation: InterfaceFlagOperation,
interface: &mut InterfaceRequest,
) -> std::io::Result<()> {
let request = match operation {
InterfaceFlagOperation::Read => libc::SIOCGIFFLAGS,
InterfaceFlagOperation::Write => libc::SIOCSIFFLAGS,
};
let request = libc::Ioctl::try_from(request).map_err(|_| {
std::io::Error::new(
std::io::ErrorKind::InvalidInput,
"interface flag ioctl request does not fit the platform ABI",
)
})?;
let interface_pointer: *mut InterfaceRequest = interface;
// SAFETY: interface points to an ifreq-compatible buffer that remains
// valid and exclusively borrowed for the duration of ioctl.
if unsafe { libc::ioctl(fd, request, interface_pointer) } < 0 {
Err(std::io::Error::last_os_error())
} else {
Ok(())
}
}
#[cfg(target_os = "linux")]
fn last_os_error(operation: &str) -> InitError {
InitError(format!("{operation}: {}", std::io::Error::last_os_error()))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn accepts_only_prepare_network_without_arguments() {
assert_eq!(
parse_command([OsString::from("prepare-network")]).expect("valid command"),
Command::PrepareNetwork
);
assert_eq!(
parse_command([OsString::from("--version")]).expect("valid version command"),
Command::Version
);
assert!(parse_command(Vec::<OsString>::new()).is_err());
assert!(parse_command([OsString::from("other")]).is_err());
assert!(
parse_command([OsString::from("prepare-network"), OsString::from("eth0")]).is_err()
);
}
#[cfg(target_os = "linux")]
#[test]
fn loopback_request_has_a_fixed_interface_name() {
let request = InterfaceRequest::loopback();
assert_eq!(request.name[0], 108);
assert_eq!(request.name[1], 111);
assert!(request.name[2..].iter().all(|byte| *byte == 0));
}
}
crates/openshell-driver-vm/src/driver.rs
+1094 -1424
View File
File diff suppressed because it is too large Load Diff
crates/openshell-driver-vm/src/embedded_runtime.rs
+1 -16
View File
@@ -10,7 +10,6 @@ use std::path::{Path, PathBuf};
mod resources {
pub const LIBKRUN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/libkrun.dylib.zst"));
pub const LIBKRUNFW: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/libkrunfw.5.dylib.zst"));
pub const GVPROXY: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/gvproxy.zst"));
pub const LIBKRUN_NAME: &str = "libkrun.dylib";
pub const LIBKRUNFW_NAME: &str = "libkrunfw.5.dylib";
}
@@ -19,7 +18,6 @@ mod resources {
mod resources {
pub const LIBKRUN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/libkrun.so.zst"));
pub const LIBKRUNFW: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/libkrunfw.so.5.zst"));
pub const GVPROXY: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/gvproxy.zst"));
pub const LIBKRUN_NAME: &str = "libkrun.so";
pub const LIBKRUNFW_NAME: &str = "libkrunfw.so.5";
}
@@ -28,7 +26,6 @@ mod resources {
mod resources {
pub const LIBKRUN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/libkrun.so.zst"));
pub const LIBKRUNFW: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/libkrunfw.so.5.zst"));
pub const GVPROXY: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/gvproxy.zst"));
pub const LIBKRUN_NAME: &str = "libkrun.so";
pub const LIBKRUNFW_NAME: &str = "libkrunfw.so.5";
}
@@ -41,7 +38,6 @@ mod resources {
mod resources {
pub const LIBKRUN: &[u8] = &[];
pub const LIBKRUNFW: &[u8] = &[];
pub const GVPROXY: &[u8] = &[];
pub const LIBKRUN_NAME: &str = "libkrun";
pub const LIBKRUNFW_NAME: &str = "libkrunfw";
}
@@ -82,7 +78,6 @@ pub fn ensure_runtime_extracted() -> Result<PathBuf, String> {
resources::LIBKRUNFW,
&cache_dir.join(resources::LIBKRUNFW_NAME),
)?;
extract_resource(resources::GVPROXY, &cache_dir.join("gvproxy"))?;
#[cfg(target_os = "macos")]
{
@@ -96,22 +91,13 @@ pub fn ensure_runtime_extracted() -> Result<PathBuf, String> {
fs::write(&version_marker, cache_key)
.map_err(|e| format!("write runtime marker {}: {e}", version_marker.display()))?;
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt as _;
fs::set_permissions(cache_dir.join("gvproxy"), fs::Permissions::from_mode(0o755))
.map_err(|e| format!("chmod gvproxy: {e}"))?;
}
Ok(cache_dir)
}
pub fn validate_runtime_dir(dir: &Path) -> Result<(), String> {
let libkrun = dir.join(resources::LIBKRUN_NAME);
let libkrunfw = dir.join(resources::LIBKRUNFW_NAME);
let gvproxy = dir.join("gvproxy");
for path in [&libkrun, &libkrunfw, &gvproxy] {
for path in [&libkrun, &libkrunfw] {
if !path.is_file() {
return Err(format!("missing runtime file: {}", path.display()));
}
@@ -128,7 +114,6 @@ fn runtime_cache_key() -> String {
let mut fp: u64 = 0;
for (index, chunk) in [resources::LIBKRUN, resources::LIBKRUNFW]
.into_iter()
.chain(std::iter::once(resources::GVPROXY))
.enumerate()
{
let sample = &chunk[..chunk.len().min(64)];

Some files were not shown because too many files have changed in this diff Show More