mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-17 05:31:16 +08:00
Seth Jennings
6461677c32
* feat(policy): accept numeric UIDs in sandbox process identity validation Allow run_as_user and run_as_group to be either the literal 'sandbox' or a numeric UID/GID within [1000, 2_000_000_000]. This removes the hard dependency on a baked-in 'sandbox' user in container images, enabling compute drivers to inject resolved UIDs at sandbox creation. Phase 1 of #1959. Signed-off-by: Seth Jennings <sjenning@redhat.com> * feat(supervisor): accept numeric UIDs for process identity dropping Allow run_as_user and run_as_group to be numeric UIDs/GIDs, removing the hard dependency on a baked-in 'sandbox' user in container images. Changes: - validate_sandbox_user(): accepts numeric UIDs without passwd lookup (logs OCSF event); keeps passwd check for "sandbox" name; rejects non-numeric non-sandbox strings that fail passwd lookup - prepare_filesystem(): passes numeric UIDs/GIDs directly to chown() instead of requiring a passwd entry - drop_privileges(): resolves numeric UIDs/GIDs directly via UID::from_raw / Gid::from_raw; skips initgroups when target uid matches current euid; uses guard conditions before setgid/setuid calls - session_user_and_home(): falls back to ("{uid}", "/sandbox") for numeric UIDs, avoiding a passwd lookup that will fail Re-exports MIN_SANDBOX_UID and MAX_SANDBOX_UID from openshell-policy so callers have consistent range constants. Phase 2 of #1959. Signed-off-by: Seth Jennings <sjenning@redhat.com> * feat(driver-kubernetes): resolve sandbox UID/GID from config or OpenShift SCC annotations Phase 3 of the numeric-UID plan: allow operators to specify explicit sandbox_uid/sandbox_gid in Kubernetes driver config, auto-detect from OpenShift SCC namespace annotations, and propagate resolved values to supervisor container env vars and PVC init container securityContext. Changes: - Add sandbox_uid/sandbox_gid fields to KubernetesComputeConfig - Add SANDBOX_UID/SANDBOX_GID env var constants to openshell-core - Implement resolve_sandbox_identity() to fetch namespace annotations and auto-detect OpenShift SCC UID ranges (sa.scc.uid-range) - Pass resolved UID/GID through SandboxPodParams to pod spec builder - Inject SANDBOX_UID/SANDBOX_GID env vars into supervisor container - Update PVC init container securityContext with resolved UID/GID instead of hard-coded root - Add comprehensive unit tests for resolution logic and annotation parsing (resolve_sandbox_uid, resolve_sandbox_gid, OpenShift SCC annotation parsing) Signed-off-by: Seth Jennings <sjenning@redhat.com> * feat(driver-vm): add configurable sandbox UID/GID and update docs/examples Phase 4 of the numeric-UID plan: replace hardcoded SANDBOX_UID (10001) in VM rootfs preparation with configurable sandbox_uid/sandbox_gid fields. Changes: - Add sandbox_uid/sandbox_gid to VmDriverConfig with serde derives - Pass resolved UID/GID through prepare_sandbox_rootfs_from_image_root to ensure_sandbox_guest_user which writes /etc/passwd/group/gshadow - Update BYOC Dockerfile: remove groupadd/useradd, document runtime UID injection and the ability to skip baked-in sandbox user - Update gateway-config.mdx: document sandbox_uid/sandbox_gid for both Kubernetes (with OpenShift SCC autodetection) and VM drivers - Update sandbox-compute-drivers.mdx: add Sandbox User Identity section explaining numeric UID support across all compute drivers - Update rootfs tests to use non-default UIDs, verify config passthrough Signed-off-by: Seth Jennings <sjenning@redhat.com> * code review changes * fix(supervisor): harden tests for restricted CI container environments Guard tests against CI-specific constraints: root without CAP_SETPCAP, UIDs with no /etc/passwd entry, and restricted /proc access. Signed-off-by: Seth Jennings <sjennings@nvidia.com> Signed-off-by: Seth Jennings <sjenning@redhat.com> --------- Signed-off-by: Seth Jennings <sjenning@redhat.com> Signed-off-by: Seth Jennings <sjennings@nvidia.com>
343 lines
15 KiB
Plaintext
343 lines
15 KiB
Plaintext
---
|
|
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
title: "Gateway Configuration File"
|
|
sidebar-title: "Gateway Config"
|
|
description: "Reference for the OpenShell gateway TOML configuration file (RFC 0003)."
|
|
keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Gateway, Configuration, TOML, Reference"
|
|
position: 5
|
|
---
|
|
|
|
The OpenShell gateway reads its configuration from a TOML file when `--config` or `OPENSHELL_GATEWAY_CONFIG` is set. When neither is set, the gateway reads `$XDG_CONFIG_HOME/openshell/gateway.toml` if that file exists. If no config file exists, the gateway starts from built-in defaults. Gateway process flags and gateway `OPENSHELL_*` environment variables override the file. Compute driver settings live in the driver TOML tables. See [RFC 0003](https://github.com/NVIDIA/OpenShell/blob/main/rfc/0003-gateway-configuration/README.md) for the full schema.
|
|
|
|
## Source Precedence
|
|
|
|
```text
|
|
Gateway CLI flag > gateway OPENSHELL_* env var > TOML file > built-in default
|
|
```
|
|
|
|
`database_url` is env-only. The loader rejects it when it appears in the file. When `OPENSHELL_DB_URL` is unset, the gateway stores its SQLite database under `$XDG_STATE_HOME/openshell/gateway/openshell.db`.
|
|
|
|
## Package-Managed Locations
|
|
|
|
Package-managed gateways do not require a TOML file. Create one at the package's optional config location when you need to override built-in defaults. Set `OPENSHELL_GATEWAY_CONFIG` in the launch environment to use a different file.
|
|
|
|
| Package | Optional Gateway TOML location |
|
|
|---|---|
|
|
| Homebrew | `$XDG_CONFIG_HOME/openshell/gateway.toml` when it exists, otherwise an existing Homebrew prefix config such as `/opt/homebrew/var/openshell/gateway.toml`. |
|
|
| Debian/Ubuntu | `$XDG_CONFIG_HOME/openshell/gateway.toml`, usually `~/.config/openshell/gateway.toml` for the systemd user service. |
|
|
| Fedora/RHEL RPM | `$XDG_CONFIG_HOME/openshell/gateway.toml`, usually `~/.config/openshell/gateway.toml` for the systemd user service. |
|
|
| Snap | `$SNAP_COMMON/gateway.toml`, usually `/var/snap/openshell/common/gateway.toml`. |
|
|
|
|
## Layout
|
|
|
|
The file is rooted at `[openshell]`. Gateway-wide settings live under `[openshell.gateway]`. Each compute driver owns its own `[openshell.drivers.<name>]` table. Shared keys set at gateway scope are inherited into driver tables when not overridden.
|
|
|
|
```toml
|
|
[openshell]
|
|
version = 1
|
|
|
|
[openshell.gateway]
|
|
# ... gateway-wide settings ...
|
|
|
|
[openshell.gateway.tls]
|
|
# ... gateway listener TLS ...
|
|
|
|
[openshell.gateway.oidc]
|
|
# ... JWT bearer auth ...
|
|
|
|
[openshell.drivers.kubernetes]
|
|
# ... driver-specific settings ...
|
|
```
|
|
|
|
## Full Example
|
|
|
|
A complete gateway configuration covering every section. Trim to the fields you need.
|
|
|
|
```toml
|
|
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
|
|
[openshell]
|
|
version = 1
|
|
|
|
[openshell.gateway]
|
|
bind_address = "0.0.0.0:8080"
|
|
health_bind_address = "0.0.0.0:8081"
|
|
metrics_bind_address = "0.0.0.0:9090"
|
|
|
|
log_level = "info"
|
|
|
|
# When empty, the gateway auto-detects Kubernetes, then Podman, then Docker.
|
|
# VM is never auto-detected and requires an explicit entry here.
|
|
compute_drivers = ["kubernetes"]
|
|
|
|
sandbox_namespace = "openshell"
|
|
ssh_session_ttl_secs = 3600
|
|
|
|
# Subject Alternative Names baked into the gateway server certificate.
|
|
# Wildcard DNS SANs (e.g. "*.dev.openshell.localhost") also enable sandbox
|
|
# service URLs under that domain.
|
|
server_sans = ["openshell", "*.dev.openshell.localhost"]
|
|
# Allow plaintext HTTP routing for loopback sandbox service URLs.
|
|
enable_loopback_service_http = true
|
|
|
|
# Set true only for local plaintext gateways or trusted TLS termination.
|
|
disable_tls = false
|
|
|
|
# Shared driver defaults. These inherit into [openshell.drivers.<name>] tables
|
|
# when the driver-specific table does not override them.
|
|
default_image = "ghcr.io/nvidia/openshell/sandbox:latest"
|
|
supervisor_image = "ghcr.io/nvidia/openshell/supervisor:latest"
|
|
client_tls_secret_name = "openshell-client-tls"
|
|
service_account_name = "openshell-sandbox"
|
|
host_gateway_ip = "10.0.0.1"
|
|
enable_user_namespaces = false
|
|
sa_token_ttl_secs = 3600
|
|
guest_tls_ca = "/etc/openshell/certs/ca.pem"
|
|
guest_tls_cert = "/etc/openshell/certs/client.pem"
|
|
guest_tls_key = "/etc/openshell/certs/client-key.pem"
|
|
|
|
# Optional gRPC rate limit. Both values must be positive to enable the limit.
|
|
# Set either value to 0, or omit both, to disable rate limiting.
|
|
grpc_rate_limit_requests = 120
|
|
grpc_rate_limit_window_seconds = 60
|
|
|
|
# Gateway listener TLS (distinct from the per-driver guest_tls_*).
|
|
[openshell.gateway.tls]
|
|
cert_path = "/etc/openshell/certs/gateway.pem"
|
|
key_path = "/etc/openshell/certs/gateway-key.pem"
|
|
client_ca_path = "/etc/openshell/certs/client-ca.pem"
|
|
require_client_auth = false
|
|
|
|
[openshell.gateway.gateway_jwt]
|
|
signing_key_path = "/etc/openshell/jwt/signing.pem"
|
|
public_key_path = "/etc/openshell/jwt/public.pem"
|
|
kid_path = "/etc/openshell/jwt/kid"
|
|
gateway_id = "openshell"
|
|
# Omit or set to 0 only for local single-player Docker, Podman, or VM gateways.
|
|
ttl_secs = 3600
|
|
|
|
[openshell.gateway.auth]
|
|
allow_unauthenticated_users = false
|
|
|
|
[openshell.gateway.mtls_auth]
|
|
enabled = false
|
|
|
|
[openshell.gateway.oidc]
|
|
issuer = "https://idp.example.com/realms/openshell"
|
|
audience = "openshell-cli"
|
|
jwks_ttl_secs = 3600
|
|
roles_claim = "realm_access.roles"
|
|
admin_role = "openshell-admin"
|
|
user_role = "openshell-user"
|
|
scopes_claim = ""
|
|
```
|
|
|
|
Local Docker, Podman, and VM gateways can also set `[openshell.gateway.mtls_auth] enabled = true` to authenticate CLI callers from verified client certificates. Kubernetes deployments must leave this unset and use OIDC or a trusted access proxy; the Helm chart does not render this table.
|
|
|
|
`[openshell.gateway.gateway_jwt] ttl_secs` controls gateway-minted sandbox JWT lifetime. When omitted, it defaults to `0`: the token `exp` claim and `expires_at_ms` response field become `0`, and the sandbox JWT does not expire. Use that default only for local single-player Docker, Podman, or VM gateways. Kubernetes and other shared deployments should set a positive TTL; Helm renders `3600` seconds by default, and the gateway logs a warning when a Kubernetes gateway uses `0`.
|
|
|
|
`[openshell.gateway.auth] allow_unauthenticated_users = true` is an unsafe local-development and trusted-proxy escape hatch. It accepts user-facing CLI/API calls without OIDC or mTLS credentials while sandbox supervisors still authenticate with gateway-minted sandbox JWTs. Leave it false for shared and production gateways.
|
|
|
|
`image_pull_policy` is intentionally not a shared gateway key. Kubernetes and Docker use `Always`, `IfNotPresent`, or `Never`. Podman uses `always`, `missing`, `never`, or `newer`. Set it inside the relevant driver table.
|
|
|
|
## Driver References
|
|
|
|
Each example is a complete TOML file for one compute driver. The examples repeat `[openshell]` and `[openshell.gateway]` so they stay copyable, and the driver tables list the accepted driver-specific keys. Driver-specific values override inherited gateway defaults. The gateway rejects unknown driver fields after inheritance is merged.
|
|
|
|
### Kubernetes
|
|
|
|
The gateway runs as a Pod and creates sandbox Pods in another namespace. mTLS material for sandboxes is delivered through a Kubernetes Secret rather than host-side file paths.
|
|
|
|
```toml
|
|
[openshell]
|
|
version = 1
|
|
|
|
[openshell.gateway]
|
|
bind_address = "0.0.0.0:8080"
|
|
health_bind_address = "0.0.0.0:8081"
|
|
metrics_bind_address = "0.0.0.0:9090"
|
|
log_level = "info"
|
|
compute_drivers = ["kubernetes"]
|
|
|
|
[openshell.gateway.tls]
|
|
cert_path = "/etc/openshell-tls/server/tls.crt"
|
|
key_path = "/etc/openshell-tls/server/tls.key"
|
|
client_ca_path = "/etc/openshell-tls/client-ca/ca.crt"
|
|
|
|
[openshell.drivers.kubernetes]
|
|
namespace = "agents"
|
|
service_account_name = "openshell-sandbox"
|
|
default_image = "ghcr.io/nvidia/openshell/sandbox:latest"
|
|
image_pull_policy = "IfNotPresent"
|
|
image_pull_secrets = ["regcred"]
|
|
supervisor_image = "ghcr.io/nvidia/openshell/supervisor:latest"
|
|
supervisor_image_pull_policy = "IfNotPresent"
|
|
# Use the image volume on Kubernetes >= 1.35 (GA in 1.36); switch to "init-container"
|
|
# on older clusters or where the ImageVolume feature gate is off.
|
|
supervisor_sideload_method = "image-volume"
|
|
# "combined" runs networking and process supervision in the sandbox agent
|
|
# container and preserves the existing Kubernetes sandbox behavior.
|
|
supervisor_topology = "combined"
|
|
grpc_endpoint = "https://openshell-gateway.agents.svc:8080"
|
|
ssh_socket_path = "/run/openshell/ssh.sock"
|
|
client_tls_secret_name = "openshell-client-tls"
|
|
host_gateway_ip = "10.0.0.1"
|
|
enable_user_namespaces = false
|
|
app_armor_profile = "Unconfined"
|
|
workspace_default_storage_size = "10Gi"
|
|
# Kubernetes RuntimeClass applied to sandbox pods when the API request does
|
|
# not specify one. Empty (default) = omit the field, using the cluster default.
|
|
# default_runtime_class_name = "kata-containers"
|
|
# Kubelet clamps projected tokens below 600 seconds. The driver caps values at 86400.
|
|
sa_token_ttl_secs = 3600
|
|
# Optional SPIFFE Workload API socket mounted into sandbox pods for dynamic
|
|
# provider token grants. Use an absolute path under a dedicated directory;
|
|
# shared roots such as /run, /var, /tmp, and /etc are rejected.
|
|
# Supervisor-to-gateway auth still uses gateway JWTs.
|
|
provider_spiffe_workload_api_socket_path = "/spiffe-workload-api/spire-agent.sock"
|
|
# Explicit sandbox UID/GID for the supervisor container securityContext and
|
|
# PVC init container. When unset, the driver auto-detects from OpenShift SCC
|
|
# namespace annotations (openshift.io/sa.scc.uid-range) if present, falling
|
|
# back to 1000 on non-OpenShift clusters.
|
|
# sandbox_uid = 1500
|
|
# sandbox_gid = 1500
|
|
```
|
|
|
|
### Docker
|
|
|
|
Sandboxes run as containers on a local bridge network. The supervisor binary is bind-mounted from the host (no in-cluster image pull required); guest mTLS material is supplied as host paths.
|
|
|
|
```toml
|
|
[openshell]
|
|
version = 1
|
|
|
|
[openshell.gateway]
|
|
bind_address = "127.0.0.1:17670"
|
|
log_level = "info"
|
|
compute_drivers = ["docker"]
|
|
|
|
[openshell.drivers.docker]
|
|
default_image = "ghcr.io/nvidia/openshell/sandbox:latest"
|
|
# Docker vocabulary: Always | IfNotPresent | Never. Empty behaves like IfNotPresent.
|
|
image_pull_policy = "IfNotPresent"
|
|
sandbox_namespace = "docker-dev"
|
|
# Empty auto-detects https://host.openshell.internal:<gateway-port> when guest TLS is set.
|
|
grpc_endpoint = "https://host.openshell.internal:17670"
|
|
# Skip the image-pull-and-extract step by pointing at a locally built binary.
|
|
supervisor_bin = "/usr/local/libexec/openshell/openshell-sandbox"
|
|
# When supervisor_bin is omitted, Docker extracts /openshell-sandbox from this image.
|
|
supervisor_image = "ghcr.io/nvidia/openshell/supervisor:latest"
|
|
guest_tls_ca = "/etc/openshell/certs/ca.pem"
|
|
guest_tls_cert = "/etc/openshell/certs/client.pem"
|
|
guest_tls_key = "/etc/openshell/certs/client-key.pem"
|
|
network_name = "openshell-docker"
|
|
host_gateway_ip = "172.17.0.1"
|
|
ssh_socket_path = "/run/openshell/ssh.sock"
|
|
# Unsafe operator override. Host bind mounts, including Docker local-driver
|
|
# bind-backed volumes, expose gateway-host paths inside sandboxes and can
|
|
# negate OpenShell isolation and filesystem controls.
|
|
enable_bind_mounts = false
|
|
# Set to 0 to leave Docker's runtime default unchanged.
|
|
sandbox_pids_limit = 2048
|
|
```
|
|
|
|
### Podman
|
|
|
|
Sandboxes run as Podman containers on a user-mode bridge network. The supervisor image is mounted read-only via Podman's `type=image` mount; guest mTLS material is supplied as host paths.
|
|
|
|
```toml
|
|
[openshell]
|
|
version = 1
|
|
|
|
[openshell.gateway]
|
|
bind_address = "127.0.0.1:17670"
|
|
log_level = "info"
|
|
compute_drivers = ["podman"]
|
|
|
|
[openshell.drivers.podman]
|
|
# Rootless socket path. For root Podman use /run/podman/podman.sock.
|
|
socket_path = "/run/user/1000/podman/podman.sock"
|
|
default_image = "ghcr.io/nvidia/openshell/sandbox:latest"
|
|
image_pull_policy = "missing" # always | missing | never | newer
|
|
grpc_endpoint = "https://host.containers.internal:17670"
|
|
# The gateway overwrites gateway_port from bind_address at runtime.
|
|
gateway_port = 17670
|
|
network_name = "openshell"
|
|
# Omit for the platform default: empty on Linux, 192.168.127.254 on macOS Podman machine.
|
|
# Set "" to force Podman's host-gateway resolver.
|
|
# host_gateway_ip = "192.168.127.254"
|
|
sandbox_ssh_socket_path = "/run/openshell/ssh.sock"
|
|
stop_timeout_secs = 10
|
|
supervisor_image = "ghcr.io/nvidia/openshell/supervisor:latest"
|
|
guest_tls_ca = "/etc/openshell/certs/ca.pem"
|
|
guest_tls_cert = "/etc/openshell/certs/client.pem"
|
|
guest_tls_key = "/etc/openshell/certs/client-key.pem"
|
|
# Unsafe operator override. Host bind mounts, including Podman local-driver
|
|
# bind-backed volumes, expose gateway-host paths inside sandboxes and can
|
|
# negate OpenShell isolation and filesystem controls.
|
|
enable_bind_mounts = false
|
|
# Set to 0 to leave Podman's runtime default unchanged.
|
|
sandbox_pids_limit = 2048
|
|
# Health check interval in seconds. Lower values detect readiness faster
|
|
# but increase process churn (each check spawns a conmon subprocess).
|
|
# Set to 0 to disable health checks entirely. Default: 10.
|
|
health_check_interval_secs = 10
|
|
```
|
|
|
|
### MicroVM
|
|
|
|
Each sandbox runs inside its own libkrun microVM managed by the standalone `openshell-driver-vm` subprocess. Use this driver when you want stronger isolation than container namespaces alone.
|
|
|
|
```toml
|
|
[openshell]
|
|
version = 1
|
|
|
|
[openshell.gateway]
|
|
bind_address = "127.0.0.1:17670"
|
|
log_level = "info"
|
|
# VM is never auto-detected; an explicit entry here is required.
|
|
compute_drivers = ["vm"]
|
|
|
|
[openshell.drivers.vm]
|
|
state_dir = "/var/lib/openshell/vm"
|
|
# Where the gateway looks for the openshell-driver-vm subprocess binary.
|
|
driver_dir = "/usr/local/libexec/openshell"
|
|
default_image = "ghcr.io/nvidia/openshell/sandbox:latest"
|
|
grpc_endpoint = "https://host.containers.internal:17670"
|
|
# Empty falls back to default_image.
|
|
bootstrap_image = "ghcr.io/nvidia/openshell/sandbox:latest"
|
|
krun_log_level = 1
|
|
vcpus = 2
|
|
mem_mib = 2048
|
|
overlay_disk_mib = 4096
|
|
guest_tls_ca = "/var/lib/openshell/guest-tls/ca.pem"
|
|
guest_tls_cert = "/var/lib/openshell/guest-tls/client.pem"
|
|
guest_tls_key = "/var/lib/openshell/guest-tls/client-key.pem"
|
|
# Resolved sandbox UID/GID for the rootfs /etc/passwd entry.
|
|
# Defaults to 10001 when unset; matching GID is used if sandbox_gid is empty.
|
|
# sandbox_uid = 20001
|
|
```
|
|
|
|
### Extension Driver
|
|
|
|
Extension drivers run outside the gateway and expose the
|
|
`compute_driver.proto` gRPC service on a Unix socket. Use a non-reserved driver
|
|
name; built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot
|
|
be selected through unmanaged socket endpoints. The selected driver name is the
|
|
key used for driver-owned sandbox config such as `template.driver_config.<name>`.
|
|
|
|
```toml
|
|
[openshell]
|
|
version = 1
|
|
|
|
[openshell.gateway]
|
|
bind_address = "127.0.0.1:17670"
|
|
log_level = "info"
|
|
compute_drivers = ["kyma"]
|
|
|
|
[openshell.drivers.kyma]
|
|
socket_path = "/run/openshell/kyma-compute-driver.sock"
|
|
```
|