docs(website): sync dev docs from 677875fd03

2026-11-04 08:25:55 +08:00 · 2026-10-29 00:30:32 +00:00
parent 639d4a9b2f
commit fa9aa22489
18 changed files with 581 additions and 94 deletions
@@ -1,8 +1,8 @@
 snapshots:
  dev:
-    source-ref: 229382e51dc7b0af26d5e2a845a65b3a074b6aa7
-    source-sha: 229382e51dc7b0af26d5e2a845a65b3a074b6aa7
-    version: 0.1.6.dev26
+    source-ref: 677875fd034e25906ec95c82a7ccc27ba04cf589
+    source-sha: 677875fd034e25906ec95c82a7ccc27ba04cf589
+    version: 0.1.7.dev19
  latest:
    source-ref: v0.1.6
    source-sha: ac7f615cd12d5cc49cb16eb1f5f10ffeb1e90f47
@@ -129,6 +129,22 @@ versions:
      <a href="https://docs.nvidia.com/openshell/dev/upgrade/0-1-0" target="_blank"
      rel="noreferrer">Read the 0.1.0 upgrade guide</a>.</span>'
 redirects:
+- source: /openshell/dev/kubernetes/sandbox-runtime
+  destination: /openshell/dev/kubernetes/setup
+- source: /openshell/dev/about/how-it-works
+  destination: /openshell/dev/about/architecture
+- source: /openshell/dev/extensibility/extension-negotiation
+  destination: /openshell/dev/extensibility/overview#protocol-negotiation
+- source: /openshell/dev/providers/google-vertex-ai
+  destination: /openshell/dev/how-it-works/providers/google#vertex-ai
+- source: /openshell/dev/about/supported-agents
+  destination: /openshell/dev/about/run-your-first-agent
+- source: /openshell/dev/get-started/quickstart
+  destination: /openshell/dev/about/run-your-first-agent
+- source: /openshell/dev/about/why-open-shell
+  destination: /openshell/dev/about/overview
+- source: /openshell/dev/sdk/type-script
+  destination: /openshell/dev/sdk/typescript
 - source: /openshell/latest/kubernetes/sandbox-runtime
  destination: /openshell/latest/kubernetes/setup
 - source: /openshell/latest/about/how-it-works
@@ -165,22 +181,6 @@ redirects:
  destination: /openshell/latest/tutorials
 - source: /openshell/tutorials/:path*
  destination: /openshell/latest/tutorials/:path*
- source: /openshell/dev/kubernetes/sandbox-runtime
-  destination: /openshell/dev/kubernetes/setup
- source: /openshell/dev/about/how-it-works
-  destination: /openshell/dev/about/architecture
- source: /openshell/dev/extensibility/extension-negotiation
-  destination: /openshell/dev/extensibility/overview#protocol-negotiation
- source: /openshell/dev/providers/google-vertex-ai
-  destination: /openshell/dev/how-it-works/providers/google#vertex-ai
- source: /openshell/dev/about/supported-agents
-  destination: /openshell/dev/about/run-your-first-agent
- source: /openshell/dev/get-started/quickstart
-  destination: /openshell/dev/about/run-your-first-agent
- source: /openshell/dev/about/why-open-shell
-  destination: /openshell/dev/about/overview
- source: /openshell/dev/sdk/type-script
-  destination: /openshell/dev/sdk/typescript
 - source: /openshell/:path*/index.html
  destination: /openshell/:path*
 - source: /openshell/:path*.html
@@ -91,6 +91,17 @@ The mediated channel to the **supervisor** is the workload's only allowed egress
 path. The outer fence denies everything else, so the agent can't reach a
 service, the gateway, DNS, or another private address directly.

+### Agent traces
+
+When an operator enables the
+[agent trace relay](/how-it-works/gateways/configuration#agent-trace-relay),
+the **sandbox** points the agent's OpenTelemetry SDK at the reserved address
+`192.0.0.8:4318`. Connections to it take the same mediated channel, but the
+**supervisor** answers them itself instead of opening an upstream connection.
+It tags each trace batch with the sandbox ID and forwards it to the operator's
+collector. The agent never learns the collector's address and needs no egress
+rule to reach it.
+
 ## How Each Runtime Builds the Boundary

 Every runtime follows the same contract, but each one uses the tools it already
@@ -12,7 +12,7 @@ Install OpenShell on a local workstation or Kubernetes.

 ## Install OpenShell

-Install the CLI, policy prover, and a local gateway with one command:
+On Linux or macOS, install the CLI, policy prover, and a local gateway with one command:

 ```shell
 curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh
@@ -38,7 +38,7 @@ curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh |
  OPENSHELL_VERSION=pre sh
 ```

-The installer checks prerelease tags from newest to oldest, selects an unexpired published package for the current platform, and downloads only that artifact. The selected candidate may still be qualifying or may have failed qualification. Installed packages keep the candidate's exact version, such as `0.1.0-pre.3`. Prerelease tags do not create entries on the GitHub Releases page. On Linux, prereleases and explicit release tags use Debian or RPM packages.
+The installer checks prerelease tags from newest to oldest, selects an unexpired published package for the current platform, and downloads only that artifact. The selected candidate may still be qualifying or may have failed qualification. Installed packages keep the candidate's exact version, such as `0.1.0-pre.3`. Prerelease tags do not create entries on the GitHub Releases page. On Linux, explicit release tags use Debian or RPM packages, as do prereleases by default. Set `OPENSHELL_INSTALL_METHOD=snap` with `OPENSHELL_VERSION=pre` to track the newest published prerelease snap in `latest/beta`. Beta publication runs alongside qualification, so the channel may contain a candidate that later fails qualification.

 The rolling [`dev` release](https://github.com/NVIDIA/OpenShell/releases/tag/dev) does not require GitHub authentication:

@@ -72,6 +72,21 @@ The local gateway auto-detects an available runtime. To pin one, set `compute_dr
 | Docker | Docker Desktop or Docker Engine 28.0 or later. |
 | Podman | Linux with Podman 5.x, cgroups v2, and an active Podman user socket. |
 | MicroVM | Host virtualization: Hypervisor.framework on macOS or KVM on Linux. |
+| [Windows MXC](#windows) | Native Windows x64 or ARM64 with MXC 1.0.0 and a host that passes the ProcessContainer capability probe. Development; not production-qualified. |
+
+## Windows
+
+Build native Windows x64 or ARM64 binaries from the
+[`windows` branch](https://github.com/NVIDIA/OpenShell/tree/windows).
+Follow the [Windows build and runtime guide](https://github.com/NVIDIA/OpenShell/blob/windows/WINDOWS.md)
+for MSVC prerequisites, MXC host probing, and gateway setup. Select
+`compute_driver = "mxc"` in the gateway TOML file.
+
+Native Windows uses Microsoft MXC ProcessContainer isolation. It is under
+development and is not production-qualified; see the
+[current runtime limitations](https://github.com/NVIDIA/OpenShell/blob/windows/WINDOWS.md#runtime-and-current-limits).
+The installer and release channels above do not supply native Windows binaries.
+Running Linux OpenShell inside WSL 2 is a separate, experimental option.

 ## macOS

@@ -128,6 +143,16 @@ sudo snap install openshell
 systemctl --user enable --now snap.openshell.user-gateway.service
 ```

+To install a prerelease build, select the corresponding Store channel:
+
+```shell
+sudo snap install openshell --channel=latest/beta
+# or, if already installed
+sudo snap refresh openshell --channel=latest/beta
+```
+
+`latest/beta` contains the newest prerelease published by CI and may not have passed qualification. Stable releases rebuild the selected qualified commit with the final stable version before publishing to `latest/stable`.
+
 The user gateway is disabled globally by default. The install script enables it only for the user who ran the script. After a direct snap installation, choose one user account to run the gateway and enable it with the command above. Do not enable the gateway for multiple users who may be logged in at the same time: every user gateway binds to `127.0.0.1:17670`, so only one can run on the system at a time.

 Older OpenShell snap revisions could enable the user gateway globally, causing it to start for every user. Remove that global enablement once, then enable the gateway from the account that will run it:
@@ -143,10 +168,7 @@ The global command prevents automatic activation for other users but does not st
 systemctl --user disable --now snap.openshell.user-gateway.service
 ```

-The snap installs the standalone policy prover as `openshell.prover`. The
-`openshell-prover` alias requires Snap Store approval and may not be available.
-The prover reads local policy files through the `home` interface and does not
-connect to the gateway.
+The snap installs the standalone policy prover as `openshell.prover` with an alias to `openshell-prover`. Installs of locally-built snaps will not have the alias. The prover reads local policy files through the `home` interface and does not connect to the gateway.

 The gateway runs at `https://127.0.0.1:17670`. It reads `~/snap/openshell/common/.config/openshell/gateway.toml` when that file exists. Its database defaults to `~/snap/openshell/common/.local/state/openshell/gateway/openshell.db`, and its TLS material lives under `~/snap/openshell/common/.local/state/openshell/tls`. Register it from the same user account:

@@ -55,9 +55,16 @@ OpenShell publishes multi-architecture gateway images for `linux/amd64` and `lin
 | Linux (Debian/Ubuntu)            | aarch64 (arm64)       | Supported |
 | macOS (Docker Desktop)           | Apple Silicon (arm64) | Supported |
 | Windows (WSL 2 + Docker Desktop) | x86_64                | Experimental  |
+| [Native Windows (MXC)](#windows) | x86_64 and ARM64 | `windows` branch |

 On Linux, the `openshell` CLI is a static musl binary and does not require glibc at runtime.

+### Windows
+
+Native Windows uses MXC and is separate from WSL 2. Use the
+[Windows guide](https://github.com/NVIDIA/OpenShell/blob/windows/WINDOWS.md)
+for builds from `windows`, host requirements, and enforcement limitations.
+
 ## Standalone Gateway Binary

 OpenShell publishes standalone `openshell-gateway` release assets for manual download on these platforms:
@@ -113,10 +120,7 @@ OpenShell publishes standalone `openshell-prover` release assets for manual down

 These artifacts are attached to GitHub releases. The Linux binaries are static and do not require glibc. All prover archives include the required solver linkage.

-The Debian, RPM, Homebrew, and snap packages also include the prover. Debian,
-RPM, and Homebrew installations expose it as `openshell-prover`; use
-`openshell.prover` with the snap. The `openshell-prover` snap alias requires
-Store approval and may not be available.
+The Debian, RPM, Homebrew, and snap packages also include the prover.

 ## Runtimes

@@ -31,7 +31,37 @@ to each driver crate:
 | Kubernetes | [`openshell-driver-kubernetes`](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-kubernetes/README.md) |
 | Windows MXC | [`openshell-driver-mxc`](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-mxc/README.md) |

-External compute drivers implement [`compute_driver.proto`](https://github.com/NVIDIA/OpenShell/blob/main/proto/compute_driver.proto).
+### Using an External Compute Driver
+
+External compute drivers must implement [`compute_driver.proto`](https://github.com/NVIDIA/OpenShell/blob/main/proto/compute_driver.proto).
+Refer to [Protocol Negotiation](/extensibility/overview#protocol-negotiation) for the metadata and capability requirements, and
+[Isolation Backends](/extensibility/isolation-backends) for the workload runtime
+contract. Your driver must also acknowledge the gateway's
+[resource admission policy](/how-it-works/gateways/configuration#external-resource-admission).
+
+Run your driver as a separate service listening on a Unix domain socket. The
+gateway does not start or supervise it. Restrict socket access to the gateway
+user and make the socket accessible from the gateway's host or container.
+
+Select a name that is not reserved by a built-in driver, then add it and the
+socket path to your existing `gateway.toml`:
+
+```toml
+[openshell]
+version = 2
+
+[openshell.gateway]
+compute_driver = "example-compute"
+
+[openshell.drivers.example-compute]
+socket_path = "/run/openshell/example-compute.sock"
+```
+
+Start the driver, then start or restart the gateway with this configuration. Use
+`openshell gateway info` from a CLI connected to that gateway to verify the
+active driver and negotiated capabilities. See the
+[Extension Driver configuration reference](/how-it-works/gateways/configuration#extension-driver)
+for configuration details.

 ## Credential Drivers

@@ -44,6 +44,14 @@ timeout = "500ms"
  Largest request body, response body, or WebSocket message the service inspects. Must be at most 4 MiB and no larger than the service advertises.
 </ParamField>

+<ParamField path="binding_policy" type="'dynamic' | 'allowlist' | 'exact'" default="dynamic">
+  Which advertised operation and phase pairs OpenShell uses. To require a request hook, set `binding_policy = "allowlist"` and `bindings = ["http_request_pre_credentials"]`. Both the gateway and the supervisor reject a response-only service with this registration. Keep `on_error: fail_closed` on the policy attachment to block matching requests if the middleware is unavailable.
+</ParamField>
+
+<ParamField path="bindings" type="string[]">
+  Bindings required by `allowlist` or `exact`: `http_request_pre_credentials`, `http_response_pre_return`, or `websocket_message_pre_credentials`. An allowlist ignores additional valid bindings; an exact policy rejects them. Omit this field for dynamic discovery. [Gateway Configuration](/how-it-works/gateways/configuration#supervisor-middleware-services) explains upgrade requirements.
+</ParamField>
+
 <ParamField path="timeout" type="string" default="500ms">
  Time limit for `Describe`, `ValidateConfig`, and individual evaluations or stream exchanges. Use an integer followed by `ms` or `s`, from `10ms` through `30s`. A binding may advertise a shorter evaluation timeout. Accepted HTTP response and WebSocket streams have no connection-wide deadline.
 </ParamField>
@@ -90,6 +90,9 @@ version = 2
 [openshell.gateway.oidc]
 # ... JWT bearer auth ...

+[openshell.supervisor.network]
+# ... shared network-supervisor destination trust ...
+
 [openshell.drivers.kubernetes]
 # ... driver-specific settings ...

@@ -99,6 +102,95 @@ version = 2

 The gateway selector is `compute_driver = "<driver>"`. It accepts one scalar driver name. Omit it to retain auto-detection.

+## Additional Destination CAs
+
+Use the global supervisor network table to trust private certificate authorities
+for sandbox egress destinations:
+
+```toml
+[openshell.supervisor.network]
+additional_ca_bundle = "/etc/openshell/certs/private-destination-ca.pem"
+```
+
+The path refers to one pre-assembled PEM bundle on the gateway host or in the
+gateway container. It must contain one or more PEM certificate blocks. Inline
+PEM, private keys, DER-only input, URLs, empty files, unrelated text, mixed
+certificate/key files, symlinks, and non-regular files are unsupported. The
+gateway reads the bundle once at startup, validates and canonicalizes it into a
+gateway-owned artifact, and fails startup with the field and source path when
+material is missing or unusable. It does not include certificate contents in
+diagnostics.
+
+The normalized certificates augment the default bundled, system, or native
+roots used for sandbox destination TLS. Normal certificate-chain and hostname
+verification remains enabled. A configured CA can authenticate every
+policy-permitted endpoint whose certificate chains to that CA and matches the
+requested hostname, so scope private CAs accordingly.
+
+This setting is separate from all other trust domains:
+
+- `proxy_ca_bundle` authenticates a configured corporate proxy and its TLS
+  interception path. It still requires the corresponding proxy setting.
+- `guest_tls_*`, `OPENSHELL_TLS_CA`, and Kubernetes client TLS Secrets
+  authenticate sandbox callbacks to the gateway.
+- `[openshell.gateway.tls]` configures the gateway listener.
+- OIDC, middleware, and interceptor CA settings authenticate their own
+  gateway-side connections.
+
+Docker and Podman bind-mount the gateway-owned artifact, Kubernetes creates an
+immutable content-addressed managed ConfigMap, and VM stages its snapshot in the
+sandbox overlay. Each mounts it at
+`/etc/openshell/tls/additional/ca-bundle.pem` and starts the supervisor with
+`--additional-ca-bundle` for that path. Drivers verify their staged artifact
+before handing it to a runtime; the SHA-256 generation is an internal staging
+value, not a supervisor argument. The supervisor
+reads the mounted bundle and folds it into its system roots. Remote and custom
+compute drivers reject this setting because the compute-driver protocol does
+not carry the normalized material.
+
+### Lifecycle and certificate rollover
+
+This is **global gateway configuration**, not a per-sandbox setting. Source
+files (or the Helm source ConfigMap) are read and normalized **only when the
+gateway starts**. Updating a source file or ConfigMap alone has no effect:
+OpenShell does not watch or hot-reload those sources, and it does not
+automatically restart sandbox consumers. A running supervisor retains the
+startup trust bundle it already loaded.
+
+To apply any addition, removal, or source update, the operator must:
+
+1. Update the source files or source ConfigMap.
+2. Restart or redeploy the gateway so it reads and normalizes the new source.
+3. Explicitly stop and then start **every affected sandbox**.
+
+Stop/start retains the sandbox identity and its supported durable workspace; it
+does not create a replacement sandbox. The restarted supervisor consumes the
+bundle staged from the restarted gateway's snapshot. Source normalization,
+driver delivery, and supervisor validation failures fail closed:
+they do not silently fall back to a previous or default trust bundle.
+
+Use this sequence for a non-disruptive private-CA rollover:
+
+1. Configure both the old and new roots.
+2. Restart or redeploy the gateway, then stop/start every affected sandbox.
+3. Rotate destination certificates to chain to the new root.
+4. Remove the old root, then repeat the gateway restart/redeploy and
+   stop/start sequence for every affected sandbox.
+
+For Helm, create an operator-owned ConfigMap with a `ca.crt` key and set
+`supervisor.network.additionalCaConfigMapName`. The chart mounts the source at
+`/etc/openshell-tls/additional-ca-source/ca.crt`, renders that path in
+the global table, and grants the Kubernetes driver ConfigMap `get` and `create`
+permissions for the selected workspace mode. These permissions cover the shared
+sandbox namespace in shared mode and are cluster-wide through the existing
+ClusterRoleBinding in managed and operator modes. Kubernetes RBAC cannot limit
+`create` by resource name or limit `get` to the digest-bearing name prefix. The
+chart does not grant ConfigMap `list`, `watch`, `patch`, `update`, or `delete`
+for this feature. Use a dedicated gateway service account and namespace-scoped
+bindings when this read boundary matters. Updating only the source ConfigMap
+does not trigger the gateway workload checksum, so explicitly restart or
+redeploy the gateway and then stop/start every affected sandbox.
+
 ## Migrate to schema version 2

 Schema version 2 is an intentional breaking cutover. The gateway rejects files
@@ -199,6 +291,12 @@ config_delivery_mode = "poll"
 server_sans                  = ["openshell", "*.dev.openshell.localhost"]
 # Allow plaintext HTTP routing for loopback sandbox service URLs.
 enable_loopback_service_http = true
+# Hostname format for exposed sandbox services. "hashed" makes service URLs
+# opaque one-label hashes (https://<hash>.<base-domain>/) discriminated by
+# the installation name above, instead of the readable
+# <workspace>--<sandbox>--<service> composite. Renaming the installation
+# changes every issued URL.
+service_hostname_format = "hashed"
 # Serve the WebSocket tunnel for CLI clients behind an authenticating edge proxy.
 enable_websocket_tunnel = false

@@ -223,6 +321,11 @@ provider_profile_sources = [
  { type = "user" },
 ]

+# Additional private destination roots are global supervisor settings, not
+# compute-driver settings. The gateway reads certificate-only PEM at startup.
+[openshell.supervisor.network]
+additional_ca_bundle = "/etc/openshell/certs/private-destination-ca.pem"
+
 # Operator-run supervisor middleware. The gRPC endpoint must be reachable from
 # both the gateway and sandbox supervisors.
 [[openshell.supervisor.middleware]]
@@ -325,6 +428,10 @@ certificate with an `Authorization` header. See
 [Authentication](/how-it-works/gateways/authentication#mtls-operators) for the
 certificate trust boundary.

+With operator authentication enabled and user mTLS disabled, callers without a
+verified operator certificate receive `Unauthenticated` unless another configured
+authentication mechanism accepts them or `allow_unauthenticated_users = true`.
+
 The client-certificate handshake policy has no `require_client_auth` TOML field. Client certificates are optional at the transport layer; gateway RPC authorization enforces the configured user or sandbox identity.

 `[openshell.gateway.tls]` supports optional SNI-based dual-certificate mode for deployments that need separate internal and external server certificates. Set `external_cert_path` and `external_key_path` to point at the external (e.g. ACME/publicly-trusted) certificate and key. List the hostnames that should be served with the external certificate in `external_server_names`. Connections whose TLS SNI hostname matches one of those names receive the external certificate; all other connections (including those with no SNI) receive the primary internal certificate from `cert_path`/`key_path`. Both fields must be set together — providing only one is a configuration error. On Kubernetes with the Helm chart, the external certificate is managed automatically when `certManager.serverIssuerRef.name` is set; the chart populates these fields from the cert-manager-issued external server certificate.
@@ -339,8 +446,59 @@ The gateway rejects partial or invalid signing bundles at startup, including pri

 External compute drivers declare this requirement by adding `openshell.compute.launch-authentication` to their extension metadata's `required_capabilities`. This requires a gateway that understands the launch-signing preflight and supplies `SandboxLaunchAuthentication` on creation. Drivers that use another launch mechanism can omit the requirement. The separate `supports_sandbox_authentication` capability describes the `AuthenticateSandbox` RPC and does not require launch signing.

+The gateway refuses to start before binding any ports when no OIDC issuer, mTLS user authentication, operator authentication, or `gateway_jwt` is configured and `allow_unauthenticated_users` is unset or false. Configure an authentication mechanism, or explicitly opt in for trusted local development:
+
+```toml
+[openshell.gateway.auth]
+allow_unauthenticated_users = true
+```
+
 `[openshell.gateway.auth] allow_unauthenticated_users = true` is an unsafe local-development and trusted-proxy escape hatch. It accepts user-facing CLI/API calls without OIDC or mTLS credentials while sandbox supervisors still authenticate with gateway-minted sandbox JWTs. Leave it false for shared and production gateways.

+## Service Routing
+
+Sandbox service URLs default to a readable composite hostname:
+`https://<workspace>--<sandbox>--<service>.<base-domain>/` for named services
+and `https://<workspace>--<sandbox>.<base-domain>/` for the unnamed endpoint.
+The base domain is the first wildcard DNS SAN in `server_sans`; gateways
+without SANs use the built-in `openshell.localhost`.
+
+`service_hostname_format` selects the hostname format for exposed sandbox
+services when multiple gateways share one service base domain behind a single
+ingress and wildcard certificate. Identically named workspaces, sandboxes, and
+services on two such gateways produce identical composite hostnames, so the
+shared ingress cannot select the correct gateway. Set the format to `hashed`
+with `[openshell.gateway].service_hostname_format`,
+`--service-hostname-format`, or `OPENSHELL_SERVICE_HOSTNAME_FORMAT`. The
+field defaults to `readable` (the composite hostname). Its precedence is
+`--service-hostname-format` > `OPENSHELL_SERVICE_HOSTNAME_FORMAT` > TOML >
+default.
+
+In hashed mode the discriminator hashed into every service hostname is the
+gateway installation `name` — the operator- or platform-assigned value used
+for telemetry identity. It must match `[a-z0-9]([-a-z0-9]*[a-z0-9])?` and be
+at most 63 bytes, and it must be unique among all gateways sharing the
+service base domain; that uniqueness is an operational contract enforced by
+the deploying platform, since a single gateway cannot observe its siblings.
+`gateway_jwt.gateway_id` is unrelated: it carries the sandbox JWT issuer and
+Kubernetes ownership semantics and must not be reused here.
+
+With the format set, service URLs replace the readable composite label with a
+deterministic 16-character hash of the installation name, workspace, sandbox,
+and service name: `https://<hash>.<base-domain>/` (for example,
+`https://enh47uqyj7bpyttz.apps.example.com/`, the label a gateway named
+`gateway-a` issues for workspace `default`, sandbox `my-sandbox`, service
+`web`). The hash stays one DNS label
+deep, so the existing single wildcard certificate keeps working, and it is
+unique per gateway, so exact-host ingress rules can route it unambiguously.
+The gateway still accepts requests for legacy composite hostnames while
+hashed format is configured.
+
+Service URLs are recomputed from the current configuration, not persisted.
+Changing the installation name, the hostname format, or both changes every
+hash URL issued before the change; previously issued URLs stop resolving
+until callers request a fresh URL.
+
 ## Database Pool

 `[openshell.gateway] database_max_connections` sets the connection ceiling for the gateway's database pool. One pool is shared by every database-backed RPC, so this value is also the gateway's ceiling on concurrent database work. When the pool is saturated, callers queue on connection acquisition and sqlx logs that the time to acquire a connection exceeded its slow threshold. The background readiness monitor draws from the same pool, so sustained saturation can push its database ping past the check timeout and flip `/readyz` to not-ready while the gateway is otherwise serving.
@@ -400,7 +558,7 @@ Allow enough retention and disk space for shipper outages. Graceful shutdown all

 ## OTLP Export

-`[openshell.gateway.otlp]` enables OpenTelemetry export over OTLP/gRPC. Omit the table to disable export. Its `endpoint` is the collector for every signal the gateway exports and, on its own, enables trace export. `[openshell.gateway.otlp.traces]` sends spans to a different collector. [OCSF log export](#ocsf-log-export) is off until `[openshell.gateway.otlp.ocsf]` sets `enabled = true`, and can use its own collector.
+`[openshell.gateway.otlp]` enables OpenTelemetry export over OTLP/gRPC. Omit the table to disable export. Its `endpoint` is the collector for every signal the gateway exports and, on its own, enables trace export. `[openshell.gateway.otlp.traces]` sends spans to a different collector. [OCSF log export](#ocsf-log-export) is off until `[openshell.gateway.otlp.ocsf]` sets `enabled = true`, and the [agent trace relay](#agent-trace-relay) is off until `[openshell.gateway.otlp.agent]` sets `enabled = true`. Each can use its own collector.

 The gateway already uses the Rust `tracing` framework for structured logs sent to stdout and the sandbox log stream. Enabling this section adds an OpenTelemetry layer to the same tracing subscriber. It exports span trees to an OTLP collector without exporting, replacing, or redirecting the existing diagnostic log events.

@@ -428,8 +586,8 @@ The gateway passes the trace endpoint to compute drivers and sandbox supervisors

 For Helm deployments, set `server.otlp.endpoint` to render this table. The
 optional `server.otlp.serviceName` value overrides the gateway service name;
-driver service names remain fixed. Set the `traces` and `ocsf` tables through
-`gatewayConfig`, which renders each key as a TOML table:
+driver service names remain fixed. Set the `traces`, `ocsf`, and `agent` tables
+through `gatewayConfig`, which renders each key as a TOML table:

 ```yaml
 gatewayConfig:
@@ -437,6 +595,8 @@ gatewayConfig:
    endpoint: https://traces-backend:4317
  openshell.gateway.otlp.ocsf:
    enabled: true
+  openshell.gateway.otlp.agent:
+    enabled: true
 ```

 The local `mise run helm:k3s:create` workflow installs a trace collector and UI,
@@ -487,6 +647,26 @@ A collector can accept a batch while rejecting some of its records, an OTLP part

 Metrics include `openshell_ocsf_otlp_queued_total`, `openshell_ocsf_otlp_exported_total`, `openshell_ocsf_otlp_dropped_total` (with a `reason` label of `queue_capacity`, `queue_bytes`, `oversized`, `export_failed`, `serialization`, or `closed`), `openshell_ocsf_otlp_export_errors_total`, and `openshell_ocsf_otlp_retries_total`.

+### Agent trace relay
+
+Set `enabled = true` in `[openshell.gateway.otlp.agent]` to have each sandbox forward the workload's own traces. It defaults to `false`, so trace export alone never changes the workload environment. Traces go to the table's `endpoint`, or to `[openshell.gateway.otlp]` `endpoint` when it is omitted, and that endpoint must be reachable from sandboxes.
+
+```toml
+[openshell.gateway.otlp.agent]
+enabled  = true
+endpoint = "http://agent-collector.observability.svc:4317" # optional
+```
+
+While the relay is active, the sandbox runtime sets `OTEL_EXPORTER_OTLP_ENDPOINT=http://192.0.0.8:4318` and `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` in every workload process, and sets `OTEL_EXPORTER_OTLP_COMPRESSION` and `OTEL_EXPORTER_OTLP_TRACES_COMPRESSION` to `none`, so a standard OpenTelemetry SDK exports without configuration. These are lowest-precedence defaults. They are not set when the sandbox creation request or an environment passed to `openshell sandbox exec` names `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, or when the image `ENV` names one for the main process, since `openshell sandbox exec` sessions do not inherit the image environment, and an endpoint the workload sets itself, for example with a shell `export`, takes precedence over them. A workload that exports elsewhere does so subject to the sandbox's network policy, and its traces carry no relay attribution. Provider credentials cannot redirect traces. While the relay is off, nothing is injected. Because the generic endpoint is set, an SDK that also exports metrics or logs sends them to the relay, which answers with an empty OTLP success and discards them. To keep metrics and logs, point those exporters elsewhere with `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` and `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`. The address is a reserved label rather than a network destination: the supervisor serves it inside the sandbox boundary and forwards accepted batches to the collector over OTLP/gRPC. The collector's address is never visible inside the sandbox.
+
+Every forwarded resource carries `openshell.sandbox.id`, with the same value as the supervisor's own spans, and `openshell.telemetry.source=agent`. Values the agent supplies for those two keys are replaced at the resource level; span and scope attributes pass through unchanged, so filter and join on the resource attributes. Filter on `openshell.telemetry.source` to separate agent traces from infrastructure traces, and join on `openshell.sandbox.id` to correlate them.
+
+Agents must export traces as OTLP/HTTP protobuf with compression disabled. The relay answers 415 to JSON bodies and to any `Content-Encoding` other than `identity`, 413 to bodies larger than 2 MiB, to requests with more than 512 `ResourceSpans` entries, and to requests whose resource attributes exceed 8192 in number or 256 KiB in total, 408 to requests whose body does not arrive within 30 seconds, and 503 with `Retry-After` while its buffer of 32 batches is full, which standard SDKs retry with backoff. Error responses carry a protobuf `google.rpc.Status` whose message states the reason. While the relay is off, connections to the relay address are refused immediately.
+
+Delivery is best-effort. The relay answers 200 once a batch is buffered and makes one export attempt per batch, so a batch is lost if the collector is unavailable at that moment. The supervisor logs the first failed export and a periodic summary of failures. Before the sandbox reports that its main process exited, and before a suspended sandbox's supervisor exits, the relay flushes buffered batches for up to 5 seconds, and skips that wait when its most recent export failed. It keeps serving `openshell sandbox exec` sessions after the main process exits and flushes once more when the sandbox is torn down.
+
+The relayed hop does not yet support TLS or authentication to the collector; an `https` endpoint leaves the relay inactive with a logged reason. Protect the hop at the network layer, for example with a collector-side ingress policy.
+
 ### Tuning

 `[openshell.gateway.otlp]` decides whether and where to export. How the SDK exports is controlled by the standard OpenTelemetry environment variables, which the gateway reads through the SDK rather than mirroring as TOML keys. They apply to the gateway and its in-process compute drivers. The gateway does not pass them to sandbox supervisors, which receive only the endpoints, the OCSF schema version, and the [OCSF collector headers](#ocsf-log-export).
@@ -506,6 +686,18 @@ The gateway honors these variables:
 | `OTEL_EXPORTER_OTLP_LOGS_HEADERS`, `OTEL_EXPORTER_OTLP_LOGS_COMPRESSION`, `OTEL_EXPORTER_OTLP_LOGS_TIMEOUT` | Exporter transport tuning for OCSF log records only. These override the general variables. |
 | `SSL_CERT_FILE`, `SSL_CERT_DIR` | Additional trust roots for `https://` endpoints, such as a private CA. |

+For Helm deployments, set these variables with `server.extraEnv`. Read credentials from a Secret so they stay out of the values file:
+
+```yaml
+server:
+  extraEnv:
+    - name: OTEL_EXPORTER_OTLP_HEADERS
+      valueFrom:
+        secretKeyRef:
+          name: otel-collector-auth
+          key: headers
+```
+
 These variables have no effect. Only `[openshell.gateway.otlp]` decides which signals are exported and where they go, for two reasons:

 - OCSF events are security records, so where they go must be a deliberate choice. Platform tooling can add `OTEL_*` variables to a process, as the OpenTelemetry Operator does for annotated pods on Kubernetes. Such a variable must not start sending security records to a collector, redirect them, or turn them off.
@@ -543,13 +735,30 @@ max_payload_bytes = 262144
 timeout = "500ms"
 ```

-Each service implements the supervisor middleware gRPC contract and exposes bindings through `Describe`. Policies reference the operator-owned registration `name`, attaching the complete middleware and all of its bindings. Bindings are identified by operation and phase. A manifest may expose at most one binding for each operation and phase pair. V1 supports `HttpRequest/pre_credentials`, `HttpResponse/pre_return`, and `WebSocketMessage/pre_credentials`. Registration names must be unique, and operator-run registrations cannot claim the reserved `openshell/` namespace. The service-reported manifest name is diagnostic metadata and does not need to match the registration name.
+Each service advertises operation and phase pairs through `Describe`. A sandbox policy attaches the service by its registration `name`; the gateway operator controls which bindings that attachment can use. Registration names must be unique and cannot start with the reserved `openshell/` namespace. The name in the service manifest is for diagnostics only.
+
+| `binding_policy` | What the gateway and supervisor require |
+| --- | --- |
+| `dynamic` (default) | Use all valid advertised bindings. Omit `bindings`. A service update can change which hooks run. |
+| `allowlist` | Require every entry in `bindings`. Ignore additional valid bindings the service advertises. |
+| `exact` | Require the service to advertise exactly the entries in `bindings`. |
+
+For example, add the following lines to the registration above to require a request hook even if the service also offers a response hook:
+
+```toml
+binding_policy = "allowlist"
+bindings = ["http_request_pre_credentials"]
+```
+
+The other supported bindings are `http_response_pre_return` and `websocket_message_pre_credentials`. Use each binding at most once. To require the check on matching traffic, attach the service in sandbox policy and keep the default `on_error: fail_closed` behavior.
+
+Upgrade the gateway and all supervisors before relying on `allowlist` or `exact`. Older supervisors ignore these fields. A running supervisor that rejects a configuration update keeps its last accepted configuration; confirm the new configuration is active before depending on the change.

 The gateway connects to every registered service and validates `Describe` before it starts. The service must therefore be running before the gateway. Policy creation and full policy updates call `ValidateConfig`; an unavailable service or invalid middleware configuration rejects the policy before persistence.

 Startup also requires compatible [extension protocol metadata](/extensibility/overview#protocol-negotiation). Upgrade legacy gateways and operator-run services together during a coordinated outage; neither mixed legacy/current pairing is supported. A service rejects a gateway that omits peer metadata, and a gateway rejects a service that omits metadata, uses another protocol major, or requires unsupported gateway capabilities. Protected `GetGatewayInfo` and `openshell gateway info` report the resulting non-secret snapshot.

-`max_payload_bytes` is the shared operator limit for inspectable logical payloads across every binding exposed by the service. It caps HTTP request and response units, replacement bodies, and complete WebSocket text messages and replacements. Whole-response inspection uses it as the stage's total body limit. Streaming response inspection applies it to each unit. The value must be greater than zero, no larger than each binding's advertised `max_payload_bytes` capability, and no larger than the 4 MiB platform maximum. OpenShell rejects oversized values instead of silently clamping them. Binary WebSocket messages are not exposed to V1 middleware, so this field does not limit binary pass-through. Middleware gRPC servers should allow messages of at least 4 MiB plus 293 KiB so a maximum-size payload and its protobuf envelope fit on the transport.
+`max_payload_bytes` is the shared operator limit for inspectable logical payloads across the bindings authorized for this service. It caps HTTP request and response units, replacement bodies, and complete WebSocket text messages and replacements. Whole-response inspection uses it as the stage's total body limit. Streaming response inspection applies it to each unit. The value must be greater than zero, no larger than each binding's advertised `max_payload_bytes` capability, and no larger than the 4 MiB platform maximum. OpenShell rejects oversized values instead of silently clamping them. Binary WebSocket messages are not exposed to V1 middleware, so this field does not limit binary pass-through. Middleware gRPC servers should allow messages of at least 4 MiB plus 293 KiB so a maximum-size payload and its protobuf envelope fit on the transport.

 `timeout` is the operator-configured service-wide RPC timeout. It accepts the same compact duration syntax as gateway interceptors: an integer followed by `ms` or `s`, such as `500ms` or `2s`. Values must be between `10ms` and `30s`, inclusive. Omit the field to use the 500 ms platform default. A binding may advertise a shorter `timeout` in the `Describe` manifest, but it cannot extend the operator-configured deadline; OpenShell uses the smaller value. OpenShell validates both levels before accepting the service. The operator-configured service timeout applies to `Describe` and `ValidateConfig`. The effective binding timeout applies to HTTP request evaluation, HTTP response preflight and unit exchanges, WebSocket preflight, and each WebSocket message. Accepted streaming protocols have no connection-wide RPC deadline.

@@ -1018,43 +1227,35 @@ the SPIRE OIDC discovery endpoint or its TLS CA.

 ### MXC

-The MXC driver runs Windows workloads through `wxc-exec`. Enable ETW auditing to
-map Windows Sandboxing provider events into the gateway's OCSF stream.
+The MXC driver runs native Windows workloads through `wxc-exec`. Use this
+configuration with a build from the
+[`windows` branch](https://github.com/NVIDIA/OpenShell/tree/windows). Keep
+`openshell-supervisor.exe` and `openshell-windows-sandbox.exe` beside the gateway.

 ```toml
 [openshell]
 version = 2

 [openshell.gateway]
-bind_address    = "127.0.0.1:17670"
-log_level       = "info"
+bind_address = "127.0.0.1:17670"
 compute_driver = "mxc"
+# Required when gateway TLS is enabled.
+guest_tls_ca = "C:\\OpenShell\\certs\\ca.pem"

 [openshell.drivers.mxc]
-wxc_exec_path           = "C:\\mxc-kit\\bin\\wxc-exec.exe"
-backend                 = "process_container"
-default_configuration_id = "composable"
-pc_least_privilege      = false
-pc_capabilities         = []
-debug                   = false
-etw_audit               = true
+wxc_exec_path = "C:\\mxc\\wxc-exec.exe"
+backend = "process_container"
+# Administrator opt-in for per-sandbox commands and working directories.
+allow_driver_config = true
+etw_audit = true
 ```

-`etw_audit` defaults to `false`. When enabled, the gateway account must be an
-administrator or belong to the Windows Performance Log Users group. Workload
-commands and working directories remain sandbox-scoped and must be supplied in
-the `mxc` driver configuration when creating a sandbox.
+`etw_audit` defaults to `false`. Enabling it requires an administrator account
+or membership in Windows Performance Log Users. See
+[OCSF JSON Export](/observability/ocsf-json-export) for durable audit output.

-`OPENSHELL_OCSF_JSON=1` enables the gateway's JSONL audit
-sink independently of the selected compute driver. Set `OPENSHELL_OCSF_LOG_DIR`
-to override the default `%PROGRAMDATA%\OpenShell\logs` directory on Windows
-or `$XDG_STATE_HOME/openshell/logs` on Unix. Prefer the gateway's configured
-`ocsf_log` output for retention controls and loss metrics.
-
-The driver records executable identity in process audit events and omits raw
-command arguments because they can contain credentials or personal data. See
-[OCSF JSON Export](/observability/ocsf-json-export) for durable Windows audit
-output.
+Follow the [Windows configuration guide](https://github.com/NVIDIA/OpenShell/blob/windows/docs/how-it-works/gateways/configuration.mdx#mxc)
+for host prerequisites, TLS setup, sandbox creation, and runtime limitations.

 ### Docker

@@ -1297,22 +1498,6 @@ health_check_interval_secs = 10
 # proxy_connect_by_hostname = true
 # Corporate CA trusted for an https:// proxy and TLS-intercepting proxies.
 # proxy_ca_bundle           = "/etc/openshell/tls/proxy-ca.pem"
-# additional_ca_bundle points at a PEM CA bundle (on the gateway host) trusted
-# for direct, policy-inspected HTTPS egress to a destination signed by a
-# private CA — independent of proxy_ca_bundle above, this applies whether or
-# not https_proxy is set. The Podman supervisor runs as its own container,
-# separate from the workload image, so it cannot inherit a private CA baked
-# into the workload's own trust store; this setting restores that trust
-# without rebuilding the supervisor image. The gateway bind-mounts the file
-# read-only into the supervisor container and folds it into the upstream
-# trust store alongside (not instead of) the system CA bundle. The CA is also
-# added to the trust bundle inside each sandbox, so workloads trust it directly.
-# It is trusted for every destination, not only the one that needs it, so use a
-# CA you control or one with name constraints. The file must
-# exist and contain at least one certificate; the gateway validates this on
-# the gateway host when the Podman driver starts, so a bad path or a
-# certificate-free bundle fails gateway startup rather than a later sandbox.
-# additional_ca_bundle      = "/etc/openshell/tls/additional-ca.pem"
 # Project a host Workload API Unix socket into the supervisor, or use an
 # explicit container-reachable TCP endpoint, for provider token exchange.
 # provider_spiffe_workload_api_socket = "/run/spire/agent.sock"
@@ -206,21 +206,72 @@ openshell gateway add https://127.0.0.1:8080 --local --name local

 ## Using Podman

-Replace `docker` with `podman` in the commands above. Mount the Podman socket instead of the Docker socket and set the driver to `podman`:
+The Podman driver runs sandboxes as rootless containers through the user Podman socket, and [Sandbox Runtimes](/how-it-works/sandboxes/runtimes#podman-driver) lists its requirements. The commands in this section assume rootless Podman. Enable the socket for your user:
+
+```shell
+systemctl --user enable --now podman.socket
+```
+
+Running the gateway as a rootless Podman container needs five settings beyond the Docker commands above:
+
+1. **Writable state directory.** The image runs as user `1000` and cannot write to its default home directory. Mount a host directory, set `HOME` and `XDG_DATA_HOME` to it, and add `--userns=keep-id:uid=1000,gid=1000` so the container can read and write it.
+
+2. **Same-path data directory.** The driver bind-mounts files from the data directory into each sandbox, and the host Podman resolves the path on the host. Mount the directory at the same absolute path inside the container. A named volume does not work.
+
+3. **Signing keys and TLS certificates.** Generate them with `generate-certs` and set `OPENSHELL_LOCAL_TLS_DIR` to the output directory. The gateway then serves HTTPS and requires client certificates.
+
+4. **SELinux type.** On SELinux hosts, add `--security-opt label=type:container_runtime_t`. Without it, the container cannot read the data directory or connect to the Podman socket. Podman ignores the option on other hosts. The gateway can start containers as your user through that socket, so run only the published gateway image.
+
+5. **Matching ports.** Supervisor containers call back to `127.0.0.1` on the gateway's in-container port. Publish the same port number on the host, or set `grpc_endpoint` as described below.
+
+Generate the keys and certificates:
+
+```shell
+data_dir="$HOME/.local/share/openshell-gateway"
+mkdir -p "$data_dir"
+
+podman run --rm \
+  --userns=keep-id:uid=1000,gid=1000 \
+  --security-opt label=type:container_runtime_t \
+  -v "$data_dir:$data_dir" \
+  -e HOME="$data_dir" \
+  ghcr.io/nvidia/openshell/gateway:latest \
+  generate-certs --output-dir "$data_dir/tls"
+```
+
+Start the gateway:

 ```shell
 podman run -d \
  --name openshell-gateway \
+  --userns=keep-id:uid=1000,gid=1000 \
+  --security-opt label=type:container_runtime_t \
  -p 127.0.0.1:8080:8080 \
-  -v openshell-state:/var/openshell \
+  -v "$data_dir:$data_dir" \
  -v "$XDG_RUNTIME_DIR/podman/podman.sock:/var/run/podman.sock" \
+  -e HOME="$data_dir" \
+  -e XDG_DATA_HOME="$data_dir" \
+  -e OPENSHELL_LOCAL_TLS_DIR="$data_dir/tls" \
  -e OPENSHELL_COMPUTE_DRIVER=podman \
  -e OPENSHELL_PODMAN_SOCKET=/var/run/podman.sock \
-  -e OPENSHELL_DB_URL=sqlite:/var/openshell/openshell.db \
-  -e OPENSHELL_DISABLE_TLS=true \
+  -e "OPENSHELL_DB_URL=sqlite:$data_dir/gateway.db" \
  ghcr.io/nvidia/openshell/gateway:latest
 ```

+If the gateway container exits at startup with `--tls-cert is required when TLS is enabled`, the gateway cannot read its certificates. `OPENSHELL_LOCAL_TLS_DIR` is unset, or it points at a directory that is empty or that the container cannot read. Run `generate-certs`, set the variable, and check `--userns=keep-id:uid=1000,gid=1000` and, on SELinux hosts, `--security-opt label=type:container_runtime_t`. Disabling TLS, as the error message suggests, would remove authentication.
+
+The gateway takes a few seconds to start listening. Register it with the CLI and confirm the connection:
+
+```shell
+OPENSHELL_LOCAL_TLS_DIR="$data_dir/tls" \
+  openshell gateway add https://127.0.0.1:8080 --local --name local
+openshell status
+```
+
+The CLI imports its client certificates from `OPENSHELL_LOCAL_TLS_DIR`. Without it, the CLI reads `~/.local/state/openshell/tls`, which can hold another gateway's certificates and fails with `BadSignature`. If a gateway named `local` is already registered, run `openshell gateway remove local` first.
+
+To publish a different host port, set `grpc_endpoint` to the host address under `[openshell.drivers.podman]` in a gateway TOML file. For example, with `-p 127.0.0.1:9090:8080`, the file needs only `version = 2` under `[openshell]` and `grpc_endpoint = "https://127.0.0.1:9090"`. Save it as `$data_dir/gateway.toml`, add `-e OPENSHELL_GATEWAY_CONFIG="$data_dir/gateway.toml"` to the gateway command, and register the gateway at `https://127.0.0.1:9090`. The [gateway configuration reference](/how-it-works/gateways/configuration#podman) lists the other Podman options.
+
 ## Next Steps

 - To prepare an image and launch an agent, refer to [Run Your First Agent](/about/run-your-first-agent).
@@ -42,7 +42,7 @@ For driver setup, including Docker, Podman, MicroVM, and Kubernetes paths, refer

 Sandbox service routing is enabled for gateways by default. Users expose long-running sandbox services with `openshell service expose`, and the gateway routes browser traffic from the printed service URL to the loopback port inside the sandbox.

-Loopback gateways use `openshell.localhost` service URLs. OpenShell prints a URL in the form `http://<sandbox>.openshell.localhost:<port>/` or `http://<sandbox>--<service>.openshell.localhost:<port>/` when a service name is provided.
+Loopback gateways use `openshell.localhost` service URLs. OpenShell prints a URL in the form `http://<workspace>--<sandbox>.openshell.localhost:<port>/`, or `http://<workspace>--<sandbox>--<service>.openshell.localhost:<port>/` when a service name is provided.

 Browser traffic enters the same gateway listener as mTLS-protected gRPC, but plaintext HTTP is accepted only from loopback clients and only for sandbox service hostnames. Gateway APIs, auth routes, health endpoints, and non-service hostnames remain unavailable over plaintext HTTP. Cross-origin and sibling-subdomain browser requests are rejected before reaching the sandbox service.

@@ -50,6 +50,8 @@ Disable the local browser path with `--enable-loopback-service-http=false` or `O

 Custom HTTPS service domains use the gateway server SAN configuration. Add a wildcard DNS SAN such as `*.apps.example.com` to the gateway certificate and pass the same SAN to the gateway with `--server-san` or `OPENSHELL_SERVER_SAN`.

+When multiple gateways share one service domain behind a single ingress, each gateway can set `service_hostname_format = "hashed"` (`--service-hostname-format hashed` or `[openshell.gateway].service_hostname_format`). Service URLs then use an opaque fixed-length hashed label — `https://<hash>.apps.example.com/` — derived from the gateway installation `name`, unique per gateway, instead of the readable composite hostname. See [Configuration](/how-it-works/gateways/configuration#service-routing) for the field, its validation, and the effects of changing it.
+
 For remote or non-loopback gateways, browser service URLs remain HTTPS and require normal gateway authentication.

 ## Register an Existing Gateway
@@ -36,10 +36,6 @@ contain access that the proposal risk check would flag.
 This page covers the boundary check. To learn about the proposal risk check,
 refer to [Policy Advisor](/how-it-works/policies/advisor).

-For snap installations, replace `openshell-prover` in these examples with
-`openshell.prover`. The `openshell-prover` snap alias requires Store approval
-and may not be available.
-
 The prover remains independent of the gateway at runtime. If you only need the
 standalone binary, use the artifacts listed in the
 [Support Matrix](/about/support-matrix#standalone-policy-prover). These
@@ -60,10 +56,9 @@ or MCP rules, the prover reports that it cannot check the policy instead of
 ignoring those rules. [What the Boundary Check
 Covers](#what-the-boundary-check-covers) describes each part and its limits.

-The Homebrew, Debian, RPM, and snap packages install the prover. Homebrew,
-Debian, and RPM expose it as `openshell-prover`; use `openshell.prover` with the
-snap. The CLI reads policy files on your machine, does not need a gateway, and
-does not apply or approve policies.
+The Homebrew, Debian, RPM, and snap packages install the prover. The CLI reads
+policy files on your machine, does not need a gateway, and does not apply or
+approve policies.

 Create `boundary.yaml`, a boundary that allows reading `/usr` and `/etc`:

@@ -684,6 +684,12 @@ certificate—still applies. The application is responsible for authentication
 enabled on its endpoint, and an upstream edge proxy may still apply its own
 access policy. For gateway service-domain configuration, refer to
 [Manage Gateways](/how-it-works/gateways/overview#configure-service-forwarding).
+
+Consume the URL the CLI or SDK prints rather than constructing hostnames from
+workspace, sandbox, and service names. The hostname format is a gateway
+configuration detail: gateways configured with a service routing name return
+opaque hashed hostnames (`https://<hash>.<base-domain>/`) instead of the
+readable composite.
 </Note>

 ## Monitor and Debug
@@ -21,7 +21,7 @@ the sandbox record and driver-supported persistent storage.
 | [Podman](#podman-driver) | `podman` | Rootless Linux workstations without a Docker daemon. |
 | [MicroVM](#microvm-driver) | `vm` | Workloads need a VM boundary instead of a container boundary. |
 | [Kubernetes](#kubernetes-driver) | `kubernetes` | Shared clusters, remote compute, and GPU scheduling. |
-| Windows MXC | `mxc` | Coming soon. |
+| [Windows MXC](https://github.com/NVIDIA/OpenShell/blob/windows/WINDOWS.md) | `mxc` | Native Windows workloads on x64 or ARM64. |

 ## Select a Compute Driver

@@ -3,8 +3,8 @@
 # SPDX-License-Identifier: Apache-2.0
 title: "High Availability"
 sidebar-title: "High Availability"
-description: "Run multiple OpenShell gateway replicas on Kubernetes with shared PostgreSQL, authenticated peer routing, capacity metrics, and optional autoscaling."
-keywords: "Generative AI, Cybersecurity, Kubernetes, High Availability, HA, Gateway, PostgreSQL, Replicas, Failover, Autoscaling, HPA, Metrics"
+description: "Run multiple OpenShell gateway replicas on Kubernetes with shared PostgreSQL, authenticated peer routing, capacity metrics, optional autoscaling, and an experimental multi-tenant supervisor."
+keywords: "Generative AI, Cybersecurity, Kubernetes, High Availability, HA, Gateway, PostgreSQL, Replicas, Failover, Autoscaling, HPA, Metrics, Shared Supervisor, Multi-tenant Supervisor"
 position: 3
 ---

@@ -504,6 +504,113 @@ during the upgrade, the older replicas serialize every mutation across the
 fleet. Mutations can be slower and can briefly fail under load until the
 rollout finishes.

+## (Experimental) Multi-tenant supervisor
+
+By default, the Kubernetes driver runs a supervisor Pod for each sandbox. With
+the multi-tenant topology, a shared supervisor Deployment hosts many sandboxes,
+and each sandbox keeps its own policy and identity. Supervisor replicas are
+independent of the gateway replica count.
+
+Enable the topology in your release values. Run at least two supervisor
+replicas so sandboxes can fail over:
+
+```yaml
+gatewayConfig:
+  openshell.drivers.kubernetes:
+    # Creates a supervisor Deployment alongside the gateway.
+    # Per-workspace and per-node supervisor options are planned.
+    supervisor_topology: gateway
+
+supervisor:
+  replicas: 2
+  affinity:
+    podAntiAffinity:
+      requiredDuringSchedulingIgnoredDuringExecution:
+        - labelSelector:
+            matchLabels:
+              app.kubernetes.io/name: openshell-supervisor
+              app.kubernetes.io/instance: openshell
+              app.kubernetes.io/component: supervisor
+          topologyKey: kubernetes.io/hostname
+```
+
+The chart creates the supervisor Deployment and connects it to the gateway.
+Tune the pool through the top-level `supervisor` values.
+
+The chart sets no supervisor affinity by default. The example rule places each
+supervisor replica on a different node, so losing one node leaves another
+supervisor to take over. Replace `openshell` in `app.kubernetes.io/instance`
+with your Helm release name. A required rule leaves a replica Pending when there
+are not enough nodes; use `preferredDuringSchedulingIgnoredDuringExecution` to
+spread replicas when possible without blocking scheduling.
+
+The chart also creates a PodDisruptionBudget that lets voluntary disruptions,
+such as node drains, evict one supervisor at a time. If maintenance must evict
+more supervisors at once, set `supervisor.podDisruptionBudget.enabled` to
+`false` for the maintenance window and set it back to `true` afterward:
+
+```shell
+helm upgrade openshell <chart> -n openshell --reuse-values --set supervisor.podDisruptionBudget.enabled=false
+```
+
+### Recover from a Supervisor Failure
+
+When a supervisor fails, the gateway moves its sandboxes to a live supervisor.
+The sandboxes keep their running processes and in-memory state. They pause
+briefly during the move, and open proxied connections and interactive sessions
+must reconnect. Recovery does not restore a sandbox whose own Pod or node is
+lost.
+
+If no supervisor takes over within `supervisor.recoveryTimeoutSeconds`, the
+sandbox terminates its workload. The timeout defaults to `60` seconds and
+accepts values from `30` to `300`.
+
+### Scale Supervisors Manually
+
+Change `supervisor.replicas` to scale the pool:
+
+```shell
+helm upgrade openshell <chart> -n openshell --reuse-values --set supervisor.replicas=2
+helm upgrade openshell <chart> -n openshell --reuse-values --set supervisor.replicas=1
+```
+
+Scaling up adds capacity for new sandboxes without moving existing ones.
+Scaling down hands the removed supervisor's sandboxes to the
+remaining supervisors, where they keep running and their open connections must
+reconnect.
+
+### Autoscale shared supervisors by memory
+
+Set `supervisor.autoscaling.enabled` to let a HorizontalPodAutoscaler manage
+the supervisor replica count. Your cluster must provide the Kubernetes resource
+metrics API, usually through Metrics Server.
+
+```yaml
+gatewayConfig:
+  openshell.drivers.kubernetes:
+    supervisor_topology: gateway
+
+supervisor:
+  autoscaling:
+    enabled: true
+    minReplicas: 2
+    maxReplicas: 4
+    targetMemoryAverageValue: 512Mi
+```
+
+Autoscaling is off by default. The replica limits and memory target shown are
+the defaults. The target is an absolute average memory usage per supervisor
+Pod. When enabled, the HPA owns the replica count and the chart ignores
+`supervisor.replicas`. Scaling up adds room for new sandboxes but does
+not rebalance existing ones.
+
+Enabling the HPA on an existing release can briefly reset the supervisor
+Deployment to one Pod, so enable it at install time or during a maintenance
+window. If your release uses older values, upgrade with
+`--reset-then-reuse-values` as described in
+[Autoscale with a HorizontalPodAutoscaler](#autoscale-with-a-horizontalpodautoscaler).
+When you disable the HPA, set `supervisor.replicas` in the same upgrade.
+
 ## Next Steps

 - To expose the gateway through a highly available data path, refer to
@@ -257,7 +257,7 @@ helm upgrade <release-name> oci://ghcr.io/nvidia/openshell/helm-chart \
  --set grpcRoute.replicaRouting.enabled=true
 ```

-The chart renders one Service per replica and a GRPCRoute rule that matches the `x-openshell-replica` request header. The CLI sets that header from the owner the gateway reports and retries without it if the named replica is unavailable. Replica routing requires `workload.kind=statefulset`, `workload.allowMultiReplicaStatefulSet=true`, and at most 15 replicas.
+The chart renders one Service per replica and a GRPCRoute rule that matches the `x-openshell-replica` request header. The CLI and the Rust SDK set that header from the owner the gateway reports and retry without it if the named replica is unavailable. Replica routing requires `workload.kind=statefulset`, `workload.allowMultiReplicaStatefulSet=true`, and at most 15 replicas.

 If the release uses `workload.kind=deployment`, this upgrade replaces the Deployment with a StatefulSet, so every gateway pod restarts at once. The StatefulSet also requests a 1Gi PersistentVolumeClaim per replica, which stays `Pending` on clusters without a default StorageClass.

@@ -267,6 +267,50 @@ Sandbox SSH uses the gateway endpoint registered with the CLI. No separate Helm

 When a Gateway terminates TLS, register its externally reachable `https://` endpoint with the CLI. OpenShell uses that registered endpoint for `sandbox connect`, `sandbox exec`, and port forwarding, even when the gateway pod reports an internal plaintext endpoint.

+## Route Sandbox Service Traffic
+
+The chart creates only a `Gateway` and a `GRPCRoute`. Browser and WebSocket
+traffic to exposed sandbox services — the URLs printed by
+`openshell service expose` — has no chart-managed route. Operators configure
+it themselves.
+
+Service URLs use a hostname under the gateway's service base domain, which
+comes from the wildcard SAN in `pkiInitJob.serverDnsNames` or
+`certManager.serverDnsNames` (for example `*.apps.example.com`). Create an
+`HTTPRoute` or `Ingress` that matches those hostnames and forwards to the
+OpenShell gateway Service, terminating TLS on the shared wildcard certificate.
+The route must preserve the original `Host` header: the gateway parses it to
+select the sandbox service.
+
+### Multiple gateways behind one ingress
+
+When two gateways share a cluster, one ingress, and one wildcard certificate,
+identically named workspaces, sandboxes, and services produce identical
+composite hostnames, so the ingress cannot select the correct gateway. Give
+each gateway a unique routing name:
+
+```shell
+helm upgrade openshell \
+  oci://ghcr.io/nvidia/openshell/helm-chart \
+  --version <version> \
+  --namespace openshell \
+  --set server.name=gateway-a \
+  --set server.serviceHostnameFormat=hashed
+```
+
+With a routing name set, each gateway emits opaque, fixed-length hashed
+hostnames (`https://<hash>.apps.example.com/`) for its exposed services. Hash
+hostnames are deterministic and unique per gateway, so exact-host
+`HTTPRoute`/`Ingress` rules route them unambiguously with the single shared
+wildcard certificate. The hostnames exist only once services are exposed, so
+the rules must be created as services are exposed — for example by automation
+that reads `openshell service list` or the gateway API and reconciles
+exact-host routes per gateway. The chart does not create or manage these
+routes.
+
+See [Gateway configuration](/how-it-works/gateways/configuration#service-routing)
+for the routing-name field, its validation, and the effects of changing it.
+
 ## Next Steps

 Return to [Setup](/kubernetes/setup) to complete the installation.
@@ -118,6 +118,18 @@ helm upgrade --install openshell \

 The chart automatically generates PKI secrets on first install using pre-install Helm hooks. No manual secret creation is required.

+To add labels or annotations to both certificate hook Pod templates, use
+`pkiInitJob.podLabels` and `pkiInitJob.podAnnotations`. Chart-owned Pod labels
+take precedence when custom keys conflict.
+For example, disable sidecar injection for hook Pods in an Istio-injected
+namespace:
+
+```yaml
+pkiInitJob:
+  podAnnotations:
+    sidecar.istio.io/inject: "false"
+```
+
 The default Kubernetes driver puts all workspaces in one sandbox namespace. To
 have the gateway create a namespace for each workspace, follow
 [Workspace on Kubernetes](/kubernetes/workspaces).
@@ -305,6 +317,7 @@ The most commonly changed values are:
 | `server.disableTls` | Run the gateway over plaintext HTTP. Use only behind a trusted transport. |
 | `server.auth.allowUnauthenticatedUsers` | Accept user-facing calls without OIDC or mTLS credentials. Use only for trusted local development or a fully trusted access proxy. |
 | `server.enableLoopbackServiceHttp` | Enable local plaintext HTTP for loopback sandbox service URLs. Defaults to `true`. |
+| `server.serviceHostnameFormat` | Set to `hashed` to make exposed-service hostnames opaque per-gateway hash labels when multiple gateways share one service base domain behind a single ingress; the hash discriminator is the gateway installation name (`server.name`, defaulting to the release name), which must be unique among those gateways. Empty keeps the readable composite hostnames. Refer to [Ingress](/kubernetes/ingress#multiple-gateways-behind-one-ingress). |
 | `pkiInitJob.serverDnsNames` / `certManager.serverDnsNames` | Additional gateway server DNS SANs. Wildcard SANs also enable sandbox service URLs under that domain. |
 | `supervisor.sandboxRuntime.boundaryPort` | Non-privileged TLS port used between paired supervisor and sandbox Pods. |
 | `upstreamProxy` | Operator-owned corporate HTTP forward proxy for policy-approved TLS egress. Refer to [Configure a Corporate Upstream Proxy](#configure-a-corporate-upstream-proxy). |
@@ -201,6 +201,7 @@ Mutation locks:
 | `openshell_server_mutation_lock_hold_seconds` | Histogram | `scope` | Time a mutation lock was held, recorded when it is released, including requests cancelled while they held it. A conflicting acquisition times out after 10 seconds, so the gateway also logs `mutation lock held longer than the lock wait timeout` with `scope` and `held_ms` when it releases a lock held longer than that. |
 | `openshell_server_mutation_lock_timeouts_total` | Counter | `scope` | Acquisitions that waited longer than 10 seconds, including background work such as supervisor disconnect handling and provider credential refresh. API requests fail with `UNAVAILABLE` and reason `MUTATION_LOCK_TIMEOUT`, and [API Errors](/sdk/api-errors) describes how to retry them. A lock connection that PostgreSQL does not open, although at least a second of the wait remained, counts in `openshell_server_mutation_lock_errors_total` instead. |
 | `openshell_server_mutation_lock_errors_total` | Counter | `scope` | Acquisitions that failed without timing out: PostgreSQL did not open a lock connection, or a lock statement failed. API requests fail with `INTERNAL`. |
+| `openshell_server_provisioning_deadline_skips_total` | Counter | `overdue` | Times the provisioning deadline check skipped a sandbox because another operation on this replica held its locks. The check runs about once a second, waits about 500 milliseconds in total for busy locks, skips sandboxes whose locks are still busy, and retries them on the next check, so a sandbox counts once for each check it misses. In a multi-replica deployment, only the replica that runs background reconciliation records skips. A steady rate with `overdue="true"` means expiry is delayed by a global policy, settings, or platform profile change, a provider or profile change in the sandbox's workspace, or another operation on the sandbox. The gateway logs each skip at debug level with `sandbox_id`. |
 | `openshell_server_mutation_lock_connections_in_use` | Gauge | None | PostgreSQL lock-pool connections checked out on this replica: holding mutation locks, waiting for a PostgreSQL lock, or being returned to the pool, which can take up to 5 seconds. Exported only with PostgreSQL. |
 | `openshell_server_mutation_lock_connections_capacity` | Gauge | None | Lock-pool size per replica, `4` unless [`database_lock_max_connections`](/how-it-works/gateways/configuration#database-pool) sets it. Exported only with PostgreSQL. |

@@ -240,6 +241,8 @@ The labels take these values:
  `openshell_server_grpc_requests_total` uses that name for the numeric status
  code.
 - `scope` is `global`, `workspace`, or `sandbox`.
+- `overdue` is `true` when the skipped sandbox was already past its
+  provisioning deadline, otherwise `false`.

 Gauges and counters exist from startup. They start at `0`, except
 `openshell_server_relay_pending_capacity` and
@@ -147,7 +147,13 @@ For gateway-produced records, `device.os.name` and `device.os.type_id` identify

 `metadata.uid` uniquely identifies an event and stays unchanged when that record is serialized again. Use `container.uid` to associate the event with its sandbox, not `metadata.uid`. Records without a sandbox association omit the container; unknown images are omitted rather than represented by an empty image name.

-Each JSONL file line, or the payload after the stderr marker, is a complete OCSF v1.8.0 JSON object when no downgrade is configured. Here is an example of a network connection event:
+Some fields depend on the event class:
+
+- SSH Activity records name the supervisor's SSH server as `dst_endpoint`, with `svc_name: "ssh"` and the listen socket path in `name`, because the server listens on a Unix socket rather than a network address.
+- Process Activity records from the sandbox name the `openshell-sandbox` launcher as `actor.process`.
+- `connection_info` carries `direction_id` and `direction`. OpenShell reports Unknown (`0`) where it does not track the direction.
+
+Each JSONL file line, or the payload after the stderr marker, is a complete OCSF v1.8.0 JSON object when no downgrade is configured. Records validate against OCSF v1.8.0 except for the top-level `container` object, which OCSF does not define at the event level. Here is an example of a network connection event:

 ```json
 {