trending/OpenShell
1
0
Fork 0
mirror of https://github.com/NVIDIA/OpenShell.git synced 2026-10-25 06:42:52 +08:00
Code Issues Packages Projects Releases Wiki Activity

chore: merge stage one into streamed configuration branch

Browse Source 
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
This commit is contained in:
Piotr Mlocek
2026-09-15 09:04:14 -07:00
parent 13f7f06b3e 8d89651210
commit 3501e04003
120 changed files with 13373 additions and 1869 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/build-openshell-mxc-windows/SKILL.md
+9 -11
View File
@@ -115,7 +115,7 @@ The lane targets a Windows host with Visual Studio Build Tools and rustup.
| Visual C++ ARM64 tools | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Tools.ARM64 -property installationPath` | Required for native ARM64 check, build, and tests and for x64-to-ARM64 check/build. Tests always require a native runner. |
| Visual C++ ARM64 Spectre-mitigated libraries | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Runtimes.ARM64.Spectre -property installationPath` | Required by `regorus` through `msvc_spectre_libs`; the build fails when the selected MSVC toolset lacks `lib\spectre\arm64`. |
| Visual C++ Clang tools | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Llvm.Clang -property installationPath` | Provides host-native `libclang.dll` for `bindgen` and `clang-cl.exe` for ARM64 crypto dependencies such as `aws-lc-sys`. On ARM64, the wrapper uses `VC\Tools\Llvm\Arm64\bin`. |
| Visual C++ CMake tools | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.CMake.Project -property installationPath` | Provides CMake and Ninja for native dependencies. The x64-to-ARM64 path adds Ninja to `PATH`; Z3 uses an architecture-specific prebuilt release. |
| Visual C++ CMake tools | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.CMake.Project -property installationPath` | Provides CMake and Ninja for bundled Z3 and other native dependencies. The x64-to-ARM64 path adds Ninja to `PATH`; Z3 uses MSVC's Visual Studio generator. |
| Windows SDK | `where.exe rc.exe` from a Developer PowerShell | Install an SDK containing target libraries and ARM64 tools. |
| Rust via rustup | `rustc --version` | Add each target being validated: `x86_64-pc-windows-msvc` and/or `aarch64-pc-windows-msvc`. The wrapper also adds the selected target. |
| mise | `mise --version` | Used as a task runner only. |
@@ -135,8 +135,6 @@ from this skill.
| `CARGO_TARGET_DIR` | `target` under repo root | Override Cargo output location. Use a short absolute path when x64-to-ARM64 builds approach Windows path-length limits. |
| `Z3_LIBRARY_PATH_OVERRIDE` | unset | Directory containing an x64 system `libz3.lib`; not valid for ARM64. |
| `Z3_SYS_Z3_HEADER` | unset | Full `z3.h` path required with a system Z3 library. |
| `Z3_SYS_Z3_VERSION` | `4.16.0` | Pinned official prebuilt Z3 release selected by the wrapper. |
| `READ_ONLY_GITHUB_TOKEN` | unset | Optional token for the Z3 release lookup; GitHub Actions supplies `github.token`. |
| `RUSTC_WRAPPER` | inherited | The wrapper resolves an available command to an absolute path. If it is unavailable, the wrapper warns and continues without compiler caching. |
Legacy fork variables such as `OPENSHELL_UPSTREAM`,
@@ -205,7 +203,8 @@ jobs in the current mirror push run, or push a new mirrored commit. The binaries
The ARM64 check/build steps in this x64-host contract are cross-builds. The
wrapper discovers and adds host-native LLVM and Ninja to `PATH`, requires the
ARM64 compiler and Spectre-mitigated libraries, lets ARM64 crypto crates select
`clang-cl`, and downloads the official prebuilt ARM64 Z3 static library.
`clang-cl`, and builds bundled Z3 with native MSVC `cl.exe` and the Visual
Studio generator.
On ARM64 hosts, validate the native ARM64 check, build, and test path. The
wrapper rejects test targets that do not match the host architecture, so x64
@@ -312,13 +311,12 @@ Useful log files:
| `test-x86_64-pc-windows-msvc-unsupported-*.log` | Focused unsupported-driver contract output. |
| `test-aarch64-pc-windows-msvc-unsupported-*.log` | Focused native ARM64 contract output. |
The first check downloads the pinned official Z3 archive for the target
architecture through `z3-sys`. GitHub Actions authenticates the lookup with its
read-only workflow token; local users can set `READ_ONLY_GITHUB_TOKEN` if an
unauthenticated lookup is rate-limited. Cargo stores the extracted library in
its target tree, so the Windows target cache reuses it. The artifact report
computes SHA256 through .NET directly and does not rely on the `Get-FileHash`
module being available inside the mise-launched Windows PowerShell process.
The first check builds bundled Z3 from source through `z3-sys`. Cargo stores the
native build output in its target tree, so the Windows target cache reuses it.
The resulting release executables do not require `libz3.dll`. The artifact
report computes SHA256 through .NET directly and does not rely on the
`Get-FileHash` module being available inside the mise-launched Windows
PowerShell process.
## Common Fix Patterns
.agents/skills/build-openshell-mxc-windows/reference.md
+5 -5
View File
@@ -68,11 +68,11 @@ file.
For ARM64, verify the Visual Studio instance contains the ARM64 MSVC tools,
ARM64 Spectre-mitigated libraries, Clang tools, CMake tools, and a Windows SDK.
Clang supplies host-native `libclang.dll` for `bindgen` and `clang-cl.exe` for
ARM64 crypto dependencies such as `aws-lc-sys`. Native and
x64-to-ARM64 builds use the official prebuilt Z3 4.16.0 static library for the
target architecture. An x64-to-ARM64 check/build discovers and adds host-native
Ninja to `PATH`, while the crypto crates select `clang-cl`. Use a short
`CARGO_TARGET_DIR` if Windows path-length limits are reached.
ARM64 crypto dependencies such as `aws-lc-sys`. Native builds use the normal
bundled-Z3 CMake path. An x64-to-ARM64 check/build discovers and adds
host-native Ninja to `PATH`, builds bundled Z3 with native MSVC `cl.exe` and
the Visual Studio generator, and lets the crypto crates select `clang-cl`. Use
a short `CARGO_TARGET_DIR` if Windows path-length limits are reached.
## Unsupported Driver Rules
.agents/skills/helm-dev-environment/SKILL.md
+29
View File
@@ -225,6 +225,35 @@ Envoy Gateway is already installed by Skaffold (the `envoy-gateway` Helm release
service for the proxy; klipper-lb binds it to hostPort 80, reachable via the
`8080:80` load balancer port mapping.
### BackendTLSPolicy (end-to-end TLS)
To enable end-to-end TLS between the Gateway proxy and the gateway pod, add
BackendTLSPolicy values to the Helm install:
```bash
helm upgrade --install openshell deploy/helm/openshell \
--set grpcRoute.enabled=true \
--set grpcRoute.backendTLSPolicy.enabled=true \
--set server.tls.enableMtls=false \
...
```
This requires `server.tls.enableMtls=false` because ingress proxies cannot
present client certificates to the backend. The certgen hook creates a backend
CA ConfigMap from the server Secret's `ca.crt` key. With cert-manager, a
separate post-install Job polls for the cert-manager-issued certificate (up to
`pkiInitJob.timeoutSeconds`); with built-in PKI the ConfigMap is created in the
same pre-install hook. The ConfigMap is reconciled on every upgrade so CA
rotations propagate automatically.
Key Helm values:
- `grpcRoute.backendTLSPolicy.enabled`: create the BackendTLSPolicy resource
- `grpcRoute.backendTLSPolicy.caCertificateConfigMapName`: override ConfigMap name
- `grpcRoute.backendTLSPolicy.hostname`: override backend validation hostname
- `server.tls.enableMtls`: must be `false` for BackendTLSPolicy
- `pkiInitJob.timeoutSeconds`: polling duration for cert-manager mode
- `pkiInitJob.failOnTimeout`: fail install if cert-manager times out
### Keycloak OIDC
One-time setup — only needed once per cluster lifetime:
.github/workflows/windows-msvc.yml
-3
View File
@@ -56,7 +56,6 @@ jobs:
rust_target: aarch64-pc-windows-msvc
runs-on: ${{ matrix.runner }}
env:
READ_ONLY_GITHUB_TOKEN: ${{ github.token }}
SCCACHE_GHA_ENABLED: "true"
SCCACHE_GHA_VERSION: windows-msvc-${{ matrix.arch }}
SCCACHE_CLIENT_SIDE: "1"
@@ -122,7 +121,6 @@ jobs:
rust_target: aarch64-pc-windows-msvc
runs-on: ${{ matrix.runner }}
env:
READ_ONLY_GITHUB_TOKEN: ${{ github.token }}
SCCACHE_GHA_ENABLED: "true"
SCCACHE_GHA_VERSION: windows-msvc-${{ matrix.arch }}
SCCACHE_CLIENT_SIDE: "1"
@@ -180,7 +178,6 @@ jobs:
rust_target: aarch64-pc-windows-msvc
runs-on: ${{ matrix.runner }}
env:
READ_ONLY_GITHUB_TOKEN: ${{ github.token }}
SCCACHE_GHA_ENABLED: "true"
SCCACHE_GHA_VERSION: windows-msvc-${{ matrix.arch }}
SCCACHE_CLIENT_SIDE: "1"
.trivyignore.yaml
+10
View File
@@ -32,3 +32,13 @@ misconfigurations:
- "**/deployment.yaml"
statement: >-
Images come from ghcr.io/nvidia/openshell, this project's own registry.
# The Kubernetes compute driver creates its runtime infrastructure and the
# per-sandbox outer egress fence in the configured sandbox namespace.
- id: KSV-0056
paths:
- "**/role.yaml"
statement: >-
The namespace-scoped gateway role can create Services and NetworkPolicy
resources so the compute driver can connect each sandbox runtime to its
supervisor while denying direct workload egress.
CONTRIBUTING.md
+12 -17
View File
@@ -340,19 +340,14 @@ cargo build -p openshell-prover --features bundled-z3
For x86-64 and ARM64 Windows MSVC builds, use one of these Z3 paths:
- Prebuilt Z3 (the default for `windows:*` tasks): `z3-sys` downloads the
pinned Z3 4.16.0 GitHub release for the target architecture on the first
build. Cargo reuses the extracted archive from its target directory. Windows
CI authenticates the GitHub API request with `READ_ONLY_GITHUB_TOKEN` and
preserves the archive in the architecture-specific Cargo target cache. For
cold local builds, you may set `READ_ONLY_GITHUB_TOKEN` to avoid anonymous
GitHub API rate limits.
- Bundled Z3 (the default for `windows:*` tasks): `z3-sys` builds Z3 from
source and links it into the OpenShell binaries. This keeps the release
executables independent of `libz3.dll` when they are copied to another host.
- System Z3: point `Z3_LIBRARY_PATH_OVERRIDE` at the directory containing the
target-compatible MSVC Z3 library and `Z3_SYS_Z3_HEADER` at the full path to `z3.h`.
The `windows:*` tasks use this path automatically when `Z3_LIBRARY_PATH_OVERRIDE`
is set.
- Bundled Z3: for direct Cargo builds, pass `--features bundled-z3` so `z3-sys`
builds Z3 from source.
target-compatible MSVC Z3 library and `Z3_SYS_Z3_HEADER` at the full path to
`z3.h`. The `windows:*` tasks use this path automatically when
`Z3_LIBRARY_PATH_OVERRIDE` is set. Deploy the matching `libz3.dll` with any
binaries built against a dynamic system Z3.
`openshell-prover` itself has no `bindgen`/`libclang` dependency, so building
just this crate does not require `LIBCLANG_PATH`:
@@ -365,17 +360,17 @@ cargo build -p openshell-prover --target x86_64-pc-windows-msvc --features bundl
To build the full set of Windows binaries, including `openshell-gateway.exe`
and `openshell.exe`, use the `windows:build:x64` mise task instead of a
single-crate `cargo build`. It downloads the pinned prebuilt Z3 release by default. A
full build also compiles crates that use `bindgen` (e.g. the MXC driver on
Windows), so it requires `libclang.dll`; if LLVM is not on the default search
path, set `LIBCLANG_PATH` to the directory containing `libclang.dll`:
single-crate `cargo build`. It builds Z3 from source (bundled) by default. A full
build also compiles crates that use `bindgen` (e.g. the MXC driver on Windows),
so it requires `libclang.dll`; if LLVM is not on the default search path, set
`LIBCLANG_PATH` to the directory containing `libclang.dll`:
```powershell
$env:LIBCLANG_PATH='C:\Program Files\Microsoft Visual Studio\2022\<Edition>\VC\Tools\Llvm\x64\bin'
mise run --skip-tools windows:build:x64
```
To use a local x64 Z3 release instead of the prebuilt download, set
To use a local x64 Z3 release instead of the bundled build, set
`Z3_LIBRARY_PATH_OVERRIDE` and `Z3_SYS_Z3_HEADER` before running the task:
```powershell
Cargo.lock
Generated
+6
View File
@@ -3949,6 +3949,7 @@ dependencies = [
"tempfile",
"tokio",
"tracing",
"x509-parser",
]
[[package]]
@@ -4048,6 +4049,7 @@ dependencies = [
"rustls-pemfile",
"serde",
"serde_json",
"sha2 0.10.9",
"tar",
"tempfile",
"thiserror 2.0.18",
@@ -4174,6 +4176,8 @@ dependencies = [
"openshell-core",
"openshell-ocsf",
"openshell-policy",
"openshell-supervisor-network",
"rustls",
"serde",
"serde_json",
"tempfile",
@@ -4501,6 +4505,7 @@ dependencies = [
"axum",
"base64",
"bytes",
"chrono",
"clap",
"futures",
"futures-util",
@@ -5793,6 +5798,7 @@ dependencies = [
"rand 0.9.4",
"serde",
"serde_json",
"serde_yaml",
"spin",
"thiserror 2.0.18",
]
architecture/gateway.md
+24 -9
View File
@@ -341,9 +341,10 @@ public descriptor set generated by `openshell-core`; a fingerprint test in
`openshell-server` requires this inventory to be reviewed whenever it changes.
Compute-driver, credential-driver, gateway-interceptor, and
supervisor-middleware services are compiled contracts for internal extension
boundaries, not public gateway RPCs. The current public inventory has 74
methods, 291 messages, and 15 enums
(`3ba470c3278d0c7a49b7dde7ee735017604afab5054fb7a95665c71a602121f1`).
boundaries, not public gateway RPCs. The current public inventory has 75
methods, 295 messages, and 16 enums
(`d6e92d196ee88c91d13b465b2185bdcfc9858771467b06f43bce8794da84f2c9`).
`ReportEndpointStatus` is a sandbox-authenticated public gateway RPC. Its request, response, and `EndpointObservation` messages belong only to the public closure. `EndpointStatus` and `EndpointResult` also belong to the durable closure because `Sandbox.status.endpoint_statuses` persists them. The repeated status field uses a new wire tag; stored sandboxes without it decode with an empty endpoint list and retain their lifecycle fields. A fixed payload encoded with the earlier sandbox schema verifies that no database rewrite is required.
The removed `NetworkBinary.harness` field remains reserved by number and name,
so protobuf implementations cannot reuse its wire slot or source identifier.
The durable-policy compatibility decoder reads the former boolean before Prost
@@ -363,12 +364,7 @@ Go, Python, and TypeScript client generation inputs do not advertise them.
| Public messages used directly as encoded storage roots | `Sandbox`, `SandboxWorkloadTemplate`, `Provider`, `Workspace`, `WorkspaceMember`, `SshSession`, `ServiceEndpoint` | The generated public type is also the persisted payload. `SshSession` is not in the current public RPC message closure. |
| Embedded encoded root | `SandboxPolicy` | Stored in policy rows and inside the JSON settings envelope. |
The 12 encoded durable roots above have a closure of 81 messages and eight
enums (`369b36511c2e38b9df9621704a00123516c7538d8ee89a499158d7de5cee1882`).
Its intersection with the public RPC closure contains 71 messages and eight
enums (`05add438ba041defc98d791038ae593d3f09352677cae43f2276d494205ce415`).
The descriptor-derived test owns these full inventories; the tables here record
the reviewed roots and classifications.
The descriptor-derived test inventories the complete message and enum closure of the encoded durable roots and its intersection with the public RPC closure. The tables here record the reviewed roots and classifications.
| Dual-purpose encoded root | Current decision |
|---|---|
@@ -797,6 +793,25 @@ requested present -> generate and write. This guards continuity across restarts
and upgrades while still recovering cleanly if an operator deletes everything
and starts over.
When `grpcRoute.backendTLSPolicy.enabled=true`, the certgen hook also creates a
`ConfigMap` containing the CA certificate (`ca.crt`) used by the Gateway proxy
to validate the backend pod's TLS certificate. The CA is always read from the
authoritative server Secret (not the in-memory bundle) so that enabling
BackendTLSPolicy on an existing release uses the CA that actually signed the
server certificate. The ConfigMap is reconciled on every hook run: if the CA
changes (rotation, re-issue), the ConfigMap is updated in place. In built-in PKI
mode the ConfigMap is created in the same pre-install hook. In cert-manager
mode, a separate post-install/post-upgrade hook Job polls for the cert-manager-
issued server Secret and then creates or updates the ConfigMap, because
cert-manager Certificate resources are regular release objects applied after
pre-install hooks.
The `server.tls.enableMtls` value controls whether the gateway requires client
certificates. When `enableMtls` is `false`, the gateway runs HTTPS-only without
client certificate verification (use OIDC for identity instead). BackendTLSPolicy
requires `enableMtls=false` because the ingress proxy cannot present client
certificates to the backend.
Operators who manage TLS PKI with cert-manager enable `certManager.enabled`;
cert-manager takes precedence over built-in TLS generation and the chart still
renders the JWT-only hook. Operators who pre-create all TLS and JWT Secrets can
architecture/sandbox.md
+34
View File
@@ -53,6 +53,20 @@ OpenShell uses overlapping controls rather than a single sandbox primitive:
The supervisor may enrich baseline filesystem allowances for runtime-required
paths, such as proxy support files or GPU device paths when a GPU is present.
These internal allowances must stay sandbox-scoped and avoid exposing host
secrets. For example, MXC governed egress grants the generated public CA bundle
while the ephemeral CA private key remains in the host proxy's memory.
This limitation does not prevent MXC sandboxes from launching in general.
Filesystem policies work normally, and a `process_container` sandbox with
governed egress enabled can still enforce `network_policies` through the host
proxy. The limitation applies only when the effective policy contains
`network_middlewares`, which select built-in or remote services that inspect or
transform network traffic. The MXC host proxy does not currently receive the
gateway registry that resolves those services. MXC therefore rejects such a
policy synchronously during `CreateSandbox`, before it inserts runtime state or
invokes `wxc-exec`, rather than running a chain with missing implementations.
Remove the middleware entries or use a compute driver whose sandbox supervisor
receives the gateway middleware registry.
## Network and Provider Access
@@ -350,6 +364,26 @@ auth file without the acknowledgement, and the acknowledgement without an
auth file, so credentials are never sent in cleartext without an explicit
operator decision.
## Tool server connection status
A sandbox can be `Ready` while a call to an external tool server fails. For configured endpoints that use MCP over HTTP, OpenShell records the last observed network result beside the endpoint's address. Users can identify the server and distinguish policy denial, unavailable credentials, TLS or network failure, and an upstream HTTP rejection without combining client and supervisor logs. These observations do not affect sandbox lifecycle readiness.
```mermaid
flowchart LR
Traffic[Calls to configured tool servers] --> Observer[Observe network result]
Observer --> Reporter[Background reporter]
Reporter --> Gateway[Validate and store results]
Gateway --> Status[Endpoint address, last result, report time]
```
The gateway exposes one record per configured endpoint in `Sandbox.status.endpoint_statuses`. Each record contains its address, an opaque identifier, a typed result, and the time the gateway accepted that result. The address remains available when evidence resets to `NoObservedExchange`. Ordinary sandbox conditions continue to describe lifecycle and platform state.
Network observers send only the endpoint identifier and a fixed result classification, without credentials, payloads, or raw upstream errors. Requests capture observation authority before selecting policy or credentials, then bind the selected policy hash and provider revision to that capture. Observation handles also identify the installed endpoint inventory and supervisor authority, so a concurrent update cannot attribute an old request to a new configuration, and reinstalling the same configuration cannot revive an obsolete request. The sandbox reporter coalesces observations and retries an immutable snapshot through `ReportEndpointStatus`. Bounded delivery can drop observations, so endpoint status is not a complete request history.
The gateway validates the reporting supervisor's session, configuration revisions, and report sequence before atomically storing results. Global policy writes share the report's synchronization boundary. An effective policy change resets all endpoint evidence; repeated acknowledgements and metadata-only policy revisions preserve it. A provider environment change resets evidence for endpoints that depend on those credentials. Supervisor disconnection or replacement and gateway restart also invalidate observation authority. Identical report retries leave timestamps unchanged. After an inventory reset, still-valid pending evidence can be accepted again if an acknowledgement was lost, advancing the report time without another exchange.
These are passive observations with no expiry. `HttpResponseReceived` means the server returned a final HTTP status below 400, including a protocol upgrade; that response can still contain an MCP error. Informational responses alone do not establish success. MCP protocol-version and request-body policy rejections produce `PolicyDenied`. A failure before the HTTP path is known updates status only when the host and port identify one distinct endpoint. Ambiguous failures remain in structured events and logs. Results combine callers and effective ports for an endpoint. Consumers that need current tool availability must verify an actual operation. The sandbox management guide explains the public fields and results.
## Credentials
Provider credentials are stored at the gateway and fetched by the supervisor at
architecture/windows-msvc-build.md
+11 -14
View File
@@ -110,16 +110,13 @@ packaging-asset tests; its
cross-platform Python, Markdown, license, and documentation checks still run.
Test tasks require the Rust target architecture to match the Windows host, so
an ARM64 test result is native coverage rather than x64 emulation coverage.
By default it enables the `z3-sys` prebuilt-release feature and pins Z3 4.16.0.
On a clean target directory, `z3-sys` downloads the official static library for
the selected Windows architecture instead of compiling Z3 through
CMake/MSBuild. GitHub Actions supplies its read-only workflow token for the
release lookup, and the Cargo target cache preserves the extracted library for
subsequent runs. When
`Z3_LIBRARY_PATH_OVERRIDE` points at a directory containing `libz3.lib`, the
wrapper uses that system Z3 instead and requires `Z3_SYS_Z3_HEADER` to point at
the full path to `z3.h`. Local clean builds use the unauthenticated GitHub API
unless `READ_ONLY_GITHUB_TOKEN` is set.
By default it enables bundled Z3 for reproducible Windows builds. `z3-sys`
builds Z3 from source and links it into the release binaries, so copying
`openshell-gateway.exe` and `openshell.exe` does not create an undeclared
`libz3.dll` runtime dependency. When `Z3_LIBRARY_PATH_OVERRIDE` points at a
directory containing `libz3.lib`, the wrapper uses that system Z3 instead and
requires `Z3_SYS_Z3_HEADER` to point at the full path to `z3.h`. Binaries built
against a dynamic system Z3 must be deployed with the matching `libz3.dll`.
GitHub Actions layers the Cargo target cache with sccache's GitHub Actions
backend. The target cache lets Cargo skip intact dependency builds; sccache
@@ -139,10 +136,10 @@ Spectre-mitigated libraries, host-native Clang tools, CMake tools, and an
ARM64-capable Windows SDK. Clang provides `libclang.dll` for `bindgen` and
`clang-cl.exe` for ARM64 crypto dependencies. During x64-to-ARM64 check/build,
the wrapper discovers and adds the Visual Studio-bundled Ninja to `PATH` for
native dependencies. Z3 uses the official prebuilt ARM64 static library, so it
does not inherit compiler settings from those native dependencies. Artifact
hashing uses .NET SHA256 directly because module autoloading in the
mise-launched Windows PowerShell process is not guaranteed.
native dependencies. Bundled Z3 uses native MSVC `cl.exe` with the Visual
Studio generator, while the crypto crates select `clang-cl`. Artifact hashing
uses .NET SHA256 directly because module autoloading in the mise-launched
Windows PowerShell process is not guaranteed.
The wrapper defaults Cargo compilation to four jobs. Set
`OPENSHELL_WINDOWS_BUILD_JOBS` to a positive integer to override that limit.
crates/openshell-bootstrap/Cargo.toml
+1
View File
@@ -22,6 +22,7 @@ tokio = { workspace = true }
tracing = { workspace = true }
[dev-dependencies]
x509-parser = "0.16"
[lints]
workspace = true
crates/openshell-bootstrap/src/pki.rs
+58 -1
View File
@@ -3,7 +3,9 @@
use crate::jwt::{JwtKeyMaterial, generate_jwt_key};
use miette::{IntoDiagnostic, Result, WrapErr};
use rcgen::{BasicConstraints, CertificateParams, DnType, Ia5String, IsCa, KeyPair, SanType};
use rcgen::{
BasicConstraints, CertificateParams, DnType, Ia5String, IsCa, KeyPair, KeyUsagePurpose, SanType,
};
use std::net::IpAddr;
/// All PEM-encoded materials produced by [`generate_pki`].
@@ -59,6 +61,7 @@ pub fn generate_pki(extra_sans: &[String]) -> Result<PkiBundle> {
.into_diagnostic()
.wrap_err("failed to create CA params")?;
ca_params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained);
ca_params.key_usages = vec![KeyUsagePurpose::KeyCertSign, KeyUsagePurpose::CrlSign];
ca_params
.distinguished_name
.push(DnType::OrganizationName, "openshell");
@@ -80,6 +83,7 @@ pub fn generate_pki(extra_sans: &[String]) -> Result<PkiBundle> {
.into_diagnostic()
.wrap_err("failed to create server cert params")?;
server_params.subject_alt_names = server_sans;
server_params.use_authority_key_identifier_extension = true;
server_params
.distinguished_name
.push(DnType::CommonName, "openshell-server");
@@ -96,6 +100,7 @@ pub fn generate_pki(extra_sans: &[String]) -> Result<PkiBundle> {
let mut client_params = CertificateParams::new(Vec::<String>::new())
.into_diagnostic()
.wrap_err("failed to create client cert params")?;
client_params.use_authority_key_identifier_extension = true;
client_params
.distinguished_name
.push(DnType::CommonName, "openshell-client");
@@ -178,6 +183,58 @@ mod tests {
assert!(bundle.server_cert_pem.contains("BEGIN CERTIFICATE"));
}
#[test]
fn generate_pki_emits_strict_verifier_extensions() {
use x509_parser::pem::parse_x509_pem;
use x509_parser::prelude::{FromDer, ParsedExtension, X509Certificate};
let bundle = generate_pki(&[]).expect("generate_pki failed");
let parse = |pem: &str| -> Vec<u8> {
parse_x509_pem(pem.as_bytes())
.expect("valid PEM")
.1
.contents
};
let ca_der = parse(&bundle.ca_cert_pem);
let ca = X509Certificate::from_der(&ca_der).expect("valid CA cert").1;
let ca_key_usage = ca
.key_usage()
.expect("readable key usage")
.expect("CA has a key usage extension");
assert!(ca_key_usage.value.key_cert_sign());
assert!(ca_key_usage.value.crl_sign());
let ca_ski = ca
.get_extension_unique(&x509_parser::oid_registry::OID_X509_EXT_SUBJECT_KEY_IDENTIFIER)
.expect("readable SKI")
.expect("CA has a Subject Key Identifier");
for (name, pem) in [
("server", &bundle.server_cert_pem),
("client", &bundle.client_cert_pem),
] {
let der = parse(pem);
let cert = X509Certificate::from_der(&der).expect("valid leaf cert").1;
let aki = cert
.get_extension_unique(
&x509_parser::oid_registry::OID_X509_EXT_AUTHORITY_KEY_IDENTIFIER,
)
.expect("readable AKI")
.unwrap_or_else(|| panic!("{name} cert has no Authority Key Identifier"));
let (
ParsedExtension::AuthorityKeyIdentifier(aki),
ParsedExtension::SubjectKeyIdentifier(ski),
) = (aki.parsed_extension(), ca_ski.parsed_extension())
else {
panic!("{name}: unexpected extension shapes");
};
let key_id = aki
.key_identifier
.as_ref()
.unwrap_or_else(|| panic!("{name} AKI has no key identifier"));
assert_eq!(key_id.0, ski.0, "{name} AKI must match the CA SKI");
}
}
#[test]
fn build_server_sans_includes_defaults_and_extras() {
let extras = vec!["192.168.1.100".to_string(), "remote.host".to_string()];
crates/openshell-cli/src/run.rs
+336 -11
View File
@@ -46,15 +46,15 @@ use openshell_core::proto::{
ApproveAllDraftChunksRequest, ApproveDraftChunkRequest, BeginRootfsTarStagingRequest,
ClearDraftChunksRequest, CreateSandboxRequest, CreateSandboxTemplateRequest,
CreateSshSessionRequest, DeleteSandboxRequest, DeleteSandboxTemplateRequest,
DeleteServiceRequest, ExecSandboxRequest, ExposeServiceRequest, GetCurrentUserRequest,
GetDraftHistoryRequest, GetDraftPolicyRequest, GetGatewayConfigRequest,
DeleteServiceRequest, EndpointResult, EndpointStatus, ExecSandboxRequest, ExposeServiceRequest,
GetCurrentUserRequest, GetDraftHistoryRequest, GetDraftPolicyRequest, GetGatewayConfigRequest,
GetSandboxConfigRequest, GetSandboxConfigResponse, GetSandboxLogsRequest,
GetSandboxPolicyStatusRequest, GetSandboxRequest, GetSandboxTemplateRequest, GetServiceRequest,
GpuResourceRequirements, ListSandboxPoliciesRequest, ListSandboxTemplatesRequest,
ListSandboxesRequest, ListServicesRequest, PolicySource, PolicyStatus, RejectDraftChunkRequest,
ResourceRequirements, RevokeSshSessionRequest, Sandbox, SandboxPhase, SandboxPolicy,
SandboxResources, SandboxServiceLevel, SandboxSpec, SandboxStartup, SandboxTemplate,
SandboxWorkloadConfig, SandboxWorkloadTemplate, SandboxWorkloadTemplateSpec,
ResourceRequirements, RevokeSshSessionRequest, Sandbox, SandboxCondition, SandboxPhase,
SandboxPolicy, SandboxResources, SandboxServiceLevel, SandboxSpec, SandboxStartup,
SandboxTemplate, SandboxWorkloadConfig, SandboxWorkloadTemplate, SandboxWorkloadTemplateSpec,
ServiceEndpointResponse, SettingScope, StartSandboxRequest, StopSandboxRequest,
TcpForwardFrame, TcpForwardInit, TcpRelayTarget, UpdateConfigRequest, WatchSandboxRequest,
exec_sandbox_event, tcp_forward_init,
@@ -1589,6 +1589,36 @@ pub async fn sandbox_get(
);
}
if let Some(status) = &sandbox.status
&& !status.conditions.is_empty()
{
println!(" {}", "Conditions:".dimmed());
for condition in &status.conditions {
for (index, line) in sandbox_condition_display_lines(condition)
.iter()
.enumerate()
{
let prefix = if index == 0 { "- " } else { " " };
println!(" {prefix}{line}");
}
}
}
if let Some(status) = &sandbox.status
&& !status.endpoint_statuses.is_empty()
{
println!(" {}", "Tool server connections:".dimmed());
for endpoint in &status.endpoint_statuses {
for (index, line) in endpoint_status_display_lines(endpoint).iter().enumerate() {
let prefix = if index == 0 { "- " } else { " " };
println!(" {prefix}{line}");
}
}
println!(
" Results come from observed MCP over HTTP traffic. They do not check current availability or tool-call success."
);
}
let policy_from_global = config.policy_source == PolicySource::Global as i32;
println!(
" {} {}",
@@ -2392,6 +2422,29 @@ fn sandbox_to_json(sandbox: &Sandbox) -> serde_json::Value {
"resource_version": provenance.resource_version,
})
});
let conditions = sandbox.status.as_ref().map_or_else(Vec::new, |status| {
status
.conditions
.iter()
.map(sandbox_condition_to_json)
.collect::<Vec<_>>()
});
let endpoint_statuses = sandbox.status.as_ref().map_or_else(Vec::new, |status| {
status
.endpoint_statuses
.iter()
.map(|endpoint| {
serde_json::json!({
"endpoint_id": endpoint.endpoint_id,
"host": endpoint.host,
"ports": endpoint.ports,
"path": endpoint.path,
"last_result": endpoint_result_name(endpoint.last_result()),
"last_reported_at": endpoint.last_reported_at,
})
})
.collect::<Vec<_>>()
});
serde_json::json!({
"id": sandbox.object_id(),
"name": sandbox.object_name(),
@@ -2403,10 +2456,93 @@ fn sandbox_to_json(sandbox: &Sandbox) -> serde_json::Value {
"phase": phase_name(sandbox.phase()),
"current_policy_version": sandbox.current_policy_version(),
"exit_code": sandbox.status.as_ref().and_then(|status| status.exit_code),
"conditions": conditions,
"endpoint_statuses": endpoint_statuses,
"created_from_workload_template": created_from_workload_template,
})
}
fn sandbox_condition_to_json(condition: &SandboxCondition) -> serde_json::Value {
serde_json::json!({
"type": condition.r#type,
"status": condition.status,
"reason": condition.reason,
"message": condition.message,
"last_transition_time": condition.last_transition_time,
})
}
fn sandbox_condition_display_lines(condition: &SandboxCondition) -> Vec<String> {
let reason = if condition.reason.is_empty() {
String::new()
} else {
format!(" ({})", condition.reason)
};
let message = if condition.message.is_empty() {
String::new()
} else {
format!(" - {}", condition.message)
};
let mut lines = vec![format!(
"{}: {}{reason}{message}",
condition.r#type, condition.status
)];
if !condition.last_transition_time.is_empty() {
lines.push(format!(
"Last transition: {}",
condition.last_transition_time
));
}
lines
}
// These names are part of the CLI JSON/YAML contract, independent of the
// generated Rust enum's Debug output and protobuf's prefixed wire names.
fn endpoint_result_name(result: EndpointResult) -> &'static str {
match result {
EndpointResult::Unspecified => "Unspecified",
EndpointResult::NoObservedExchange => "NoObservedExchange",
EndpointResult::HttpResponseReceived => "HttpResponseReceived",
EndpointResult::PolicyDenied => "PolicyDenied",
EndpointResult::CredentialUnavailable => "CredentialUnavailable",
EndpointResult::TlsFailed => "TlsFailed",
EndpointResult::TransportFailed => "TransportFailed",
EndpointResult::UpstreamRejected => "UpstreamRejected",
}
}
fn endpoint_status_display_lines(endpoint: &EndpointStatus) -> Vec<String> {
let ports = endpoint
.ports
.iter()
.map(u32::to_string)
.collect::<Vec<_>>()
.join(", ");
let result = match endpoint.last_result() {
EndpointResult::Unspecified => "No result provided.",
EndpointResult::NoObservedExchange => "No exchange observed.",
EndpointResult::HttpResponseReceived => "Server returned an HTTP response below 400.",
EndpointResult::PolicyDenied => "Blocked by OpenShell policy.",
EndpointResult::CredentialUnavailable => {
"Required OpenShell-managed credential was unavailable."
}
EndpointResult::TlsFailed => "TLS connection failed.",
EndpointResult::TransportFailed => "Connection failed before an HTTP response arrived.",
EndpointResult::UpstreamRejected => "Server rejected the request (HTTP 400 or higher).",
};
// The gateway supplies acceptance time, which can follow the actual
// exchange. An empty timestamp means there is no accepted observation.
let reported_at = non_empty_or(&endpoint.last_reported_at, "no report yet");
vec![
format!(
"{} (ports: {ports}; path: {})",
endpoint.host, endpoint.path
),
format!("Last result: {result}"),
format!("Reported at (gateway acceptance): {reported_at}"),
]
}
fn sandbox_detail_to_json(
sandbox: &Sandbox,
config: &GetSandboxConfigResponse,
@@ -5883,12 +6019,12 @@ mod tests {
PROGRESS_STEP_STARTING_SANDBOX,
};
use openshell_core::proto::{
GetSandboxConfigResponse, GpuResourceRequirements, PolicySource, PolicyStatus,
ResourceRequirements, Sandbox, SandboxCondition, SandboxPhase, SandboxPolicy,
SandboxPolicyRevision, SandboxResources, SandboxStatus, SandboxWorkloadConfig,
SandboxWorkloadTemplate, SandboxWorkloadTemplateProvenance, SandboxWorkloadTemplateSpec,
ServiceEndpoint, ServiceEndpointResponse, WorkspaceMember, WorkspaceRole,
datamodel::v1::ObjectMeta,
EndpointResult, EndpointStatus, GetSandboxConfigResponse, GpuResourceRequirements,
PolicySource, PolicyStatus, ResourceRequirements, Sandbox, SandboxCondition, SandboxPhase,
SandboxPolicy, SandboxPolicyRevision, SandboxResources, SandboxStatus,
SandboxWorkloadConfig, SandboxWorkloadTemplate, SandboxWorkloadTemplateProvenance,
SandboxWorkloadTemplateSpec, ServiceEndpoint, ServiceEndpointResponse, WorkspaceMember,
WorkspaceRole, datamodel::v1::ObjectMeta,
};
#[test]
@@ -7144,6 +7280,195 @@ mod tests {
assert!(json["policy"].is_null());
}
#[test]
fn sandbox_detail_keeps_failed_endpoint_separate_from_ready_conditions_in_json_and_yaml() {
let sandbox = Sandbox {
status: Some(SandboxStatus {
phase: SandboxPhase::Ready as i32,
endpoint_statuses: vec![EndpointStatus {
endpoint_id: "endpoint:example".to_string(),
host: "tools.example.test".to_string(),
ports: vec![443, 8443],
path: "/mcp".to_string(),
last_result: EndpointResult::TransportFailed as i32,
last_reported_at: "2026-09-05T10:01:00Z".to_string(),
}],
conditions: vec![SandboxCondition {
r#type: "Ready".to_string(),
status: "True".to_string(),
reason: "DependenciesReady".to_string(),
message: "Supervisor session connected".to_string(),
last_transition_time: "2026-09-05T10:00:00Z".to_string(),
}],
..Default::default()
}),
..Default::default()
};
let json = super::sandbox_detail_to_json(
&sandbox,
&GetSandboxConfigResponse {
policy_source: PolicySource::Sandbox as i32,
..Default::default()
},
)
.unwrap();
// Both formats preserve the complete endpoint record and independent
// lifecycle condition without requiring a join.
let yaml = serde_yml::to_string(&json).unwrap();
let from_yaml: serde_json::Value = serde_yml::from_str(&yaml).unwrap();
for output in [json, from_yaml] {
assert_eq!(output["phase"], "Ready");
assert_eq!(
output["conditions"],
serde_json::json!([{
"type": "Ready",
"status": "True",
"reason": "DependenciesReady",
"message": "Supervisor session connected",
"last_transition_time": "2026-09-05T10:00:00Z",
}])
);
assert_eq!(
output["endpoint_statuses"],
serde_json::json!([{
"endpoint_id": "endpoint:example",
"host": "tools.example.test",
"ports": [443, 8443],
"path": "/mcp",
"last_result": "TransportFailed",
"last_reported_at": "2026-09-05T10:01:00Z",
}])
);
}
}
#[test]
fn endpoint_status_display_explains_failure_and_report_time() {
let endpoint = EndpointStatus {
endpoint_id: "endpoint:example".to_string(),
host: "tools.example.test".to_string(),
ports: vec![443, 8443],
path: "/mcp".to_string(),
last_result: EndpointResult::TransportFailed as i32,
last_reported_at: "2026-09-05T11:01:00Z".to_string(),
};
assert_eq!(
super::endpoint_status_display_lines(&endpoint),
vec![
"tools.example.test (ports: 443, 8443; path: /mcp)",
"Last result: Connection failed before an HTTP response arrived.",
"Reported at (gateway acceptance): 2026-09-05T11:01:00Z",
]
);
}
#[test]
fn endpoint_status_preserves_address_without_an_observation() {
let endpoint = EndpointStatus {
endpoint_id: "endpoint:reset".to_string(),
host: "tools.example.test".to_string(),
ports: vec![443],
path: "/**".to_string(),
last_result: EndpointResult::NoObservedExchange as i32,
last_reported_at: String::new(),
};
assert_eq!(
super::endpoint_status_display_lines(&endpoint),
vec![
"tools.example.test (ports: 443; path: /**)",
"Last result: No exchange observed.",
"Reported at (gateway acceptance): no report yet",
]
);
let sandbox = Sandbox {
status: Some(SandboxStatus {
endpoint_statuses: vec![endpoint],
..Default::default()
}),
..Default::default()
};
let output = super::sandbox_to_json(&sandbox);
assert_eq!(
output["endpoint_statuses"],
serde_json::json!([{
"endpoint_id": "endpoint:reset",
"host": "tools.example.test",
"ports": [443],
"path": "/**",
"last_result": "NoObservedExchange",
"last_reported_at": "",
}])
);
assert_eq!(output["conditions"], serde_json::json!([]));
}
#[test]
fn endpoint_status_http_response_does_not_claim_a_successful_tool_call() {
let endpoint = EndpointStatus {
host: "tools.example.test".to_string(),
ports: vec![443],
path: "/mcp".to_string(),
last_result: EndpointResult::HttpResponseReceived as i32,
last_reported_at: "2026-09-05T11:01:00Z".to_string(),
..Default::default()
};
assert_eq!(
super::endpoint_status_display_lines(&endpoint)[1],
"Last result: Server returned an HTTP response below 400."
);
}
#[test]
fn endpoint_status_structured_output_uses_stable_result_names() {
for (result, name) in [
(EndpointResult::Unspecified, "Unspecified"),
(EndpointResult::NoObservedExchange, "NoObservedExchange"),
(EndpointResult::HttpResponseReceived, "HttpResponseReceived"),
(EndpointResult::PolicyDenied, "PolicyDenied"),
(
EndpointResult::CredentialUnavailable,
"CredentialUnavailable",
),
(EndpointResult::TlsFailed, "TlsFailed"),
(EndpointResult::TransportFailed, "TransportFailed"),
(EndpointResult::UpstreamRejected, "UpstreamRejected"),
] {
let sandbox = Sandbox {
status: Some(SandboxStatus {
endpoint_statuses: vec![EndpointStatus {
last_result: result as i32,
..Default::default()
}],
..Default::default()
}),
..Default::default()
};
assert_eq!(
super::sandbox_to_json(&sandbox)["endpoint_statuses"][0]["last_result"],
name
);
}
}
#[test]
fn sandbox_condition_display_leaves_lifecycle_conditions_unchanged() {
let ordinary = SandboxCondition {
r#type: "Ready".to_string(),
status: "True".to_string(),
reason: "DependenciesReady".to_string(),
message: "Supervisor session connected".to_string(),
last_transition_time: String::new(),
};
assert_eq!(
super::sandbox_condition_display_lines(&ordinary),
vec!["Ready: True (DependenciesReady) - Supervisor session connected".to_string()]
);
}
fn log_line(
level: &str,
target: &str,
crates/openshell-cli/tests/ensure_providers_integration.rs
+9
View File
@@ -79,6 +79,15 @@ impl TestOpenShell {
#[tonic::async_trait]
impl OpenShell for TestOpenShell {
async fn report_endpoint_status(
&self,
_request: tonic::Request<openshell_core::proto::ReportEndpointStatusRequest>,
) -> Result<Response<openshell_core::proto::ReportEndpointStatusResponse>, Status> {
Ok(Response::new(
openshell_core::proto::ReportEndpointStatusResponse {},
))
}
async fn begin_rootfs_tar_staging(
&self,
_request: tonic::Request<openshell_core::proto::BeginRootfsTarStagingRequest>,
crates/openshell-cli/tests/mtls_integration.rs
+9
View File
@@ -32,6 +32,15 @@ struct TestOpenShell;
#[tonic::async_trait]
impl OpenShell for TestOpenShell {
async fn report_endpoint_status(
&self,
_request: tonic::Request<openshell_core::proto::ReportEndpointStatusRequest>,
) -> Result<Response<openshell_core::proto::ReportEndpointStatusResponse>, Status> {
Ok(Response::new(
openshell_core::proto::ReportEndpointStatusResponse {},
))
}
async fn begin_rootfs_tar_staging(
&self,
_request: tonic::Request<openshell_core::proto::BeginRootfsTarStagingRequest>,
crates/openshell-cli/tests/provider_commands_integration.rs
+9
View File
@@ -104,6 +104,15 @@ struct TestOpenShell {
#[tonic::async_trait]
impl OpenShell for TestOpenShell {
async fn report_endpoint_status(
&self,
_request: tonic::Request<openshell_core::proto::ReportEndpointStatusRequest>,
) -> Result<Response<openshell_core::proto::ReportEndpointStatusResponse>, Status> {
Ok(Response::new(
openshell_core::proto::ReportEndpointStatusResponse {},
))
}
async fn begin_rootfs_tar_staging(
&self,
_request: tonic::Request<openshell_core::proto::BeginRootfsTarStagingRequest>,
crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs
+13 -5
View File
@@ -23,11 +23,12 @@ use openshell_core::proto::{
GpuResourceRequirements, HealthRequest, HealthResponse, ListProvidersRequest,
ListProvidersResponse, ListSandboxProvidersRequest, ListSandboxProvidersResponse,
ListSandboxTemplatesRequest, ListSandboxTemplatesResponse, ListSandboxesRequest,
ListSandboxesResponse, PlatformEvent, Provider, ProviderResponse, RevokeSshSessionRequest,
RevokeSshSessionResponse, Sandbox, SandboxCondition, SandboxLogLine, SandboxPhase,
SandboxResponse, SandboxStatus, SandboxStreamEvent, SandboxTemplateResponse,
SandboxWorkloadTemplate, ServiceStatus, SettingValue, SupervisorMessage, UpdateProviderRequest,
WatchSandboxRequest, sandbox_stream_event,
ListSandboxesResponse, PlatformEvent, Provider, ProviderResponse, ReportEndpointStatusRequest,
ReportEndpointStatusResponse, RevokeSshSessionRequest, RevokeSshSessionResponse, Sandbox,
SandboxCondition, SandboxLogLine, SandboxPhase, SandboxResponse, SandboxStatus,
SandboxStreamEvent, SandboxTemplateResponse, SandboxWorkloadTemplate, ServiceStatus,
SettingValue, SupervisorMessage, UpdateProviderRequest, WatchSandboxRequest,
sandbox_stream_event,
};
use std::collections::HashMap;
use std::fs;
@@ -833,6 +834,13 @@ impl OpenShell for TestOpenShell {
Err(Status::unimplemented("not implemented in test"))
}
async fn report_endpoint_status(
&self,
_request: tonic::Request<ReportEndpointStatusRequest>,
) -> Result<Response<ReportEndpointStatusResponse>, Status> {
Err(Status::unimplemented("not implemented in test"))
}
async fn get_sandbox_logs(
&self,
_request: tonic::Request<openshell_core::proto::GetSandboxLogsRequest>,
crates/openshell-cli/tests/sandbox_name_fallback_integration.rs
+9
View File
@@ -47,6 +47,15 @@ struct TestOpenShell {
#[tonic::async_trait]
impl OpenShell for TestOpenShell {
async fn report_endpoint_status(
&self,
_request: tonic::Request<openshell_core::proto::ReportEndpointStatusRequest>,
) -> Result<Response<openshell_core::proto::ReportEndpointStatusResponse>, Status> {
Ok(Response::new(
openshell_core::proto::ReportEndpointStatusResponse {},
))
}
async fn begin_rootfs_tar_staging(
&self,
_request: tonic::Request<openshell_core::proto::BeginRootfsTarStagingRequest>,
crates/openshell-core/Cargo.toml
+1
View File
@@ -24,6 +24,7 @@ thiserror = { workspace = true }
miette = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
sha2 = { workspace = true }
tracing = { workspace = true }
url = { workspace = true }
ipnet = "2"
crates/openshell-core/src/driver_utils.rs
+1
View File
@@ -1248,6 +1248,7 @@ mod tests {
assert!(validate_guest_spiffe_tcp_endpoint("unix:/run/spire/agent.sock", true).is_err());
}
#[cfg(unix)]
#[test]
fn credential_file_rejects_fifo_without_hanging() {
// A FIFO with no writer would block a blocking open() forever. The
crates/openshell-core/src/endpoint_status.rs
+1184
View File
File diff suppressed because it is too large Load Diff
crates/openshell-core/src/grpc_client.rs
+57 -6
View File
@@ -22,13 +22,15 @@ use std::collections::HashMap;
use std::sync::{Arc, OnceLock, RwLock};
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use crate::endpoint_status::{EndpointResult, EndpointStatusSnapshot};
use crate::proto::{
DenialSummary, ExchangeProviderSubjectTokenRequest, GetDraftPolicyRequest,
GetSandboxConfigRequest, GetSandboxProviderEnvironmentRequest, IssueSandboxTokenRequest,
NetworkActivitySummary, PolicyChunk, PolicySource, PolicyStatus, RefreshSandboxTokenRequest,
ReportPolicyStatusRequest, SandboxPolicy as ProtoSandboxPolicy, SubmitPolicyAnalysisRequest,
SubmitPolicyAnalysisResponse, UpdateConfigRequest, open_shell_client::OpenShellClient,
workspace_selector,
DenialSummary, EndpointObservation as ProtoEndpointObservation,
EndpointResult as ProtoEndpointResult, ExchangeProviderSubjectTokenRequest,
GetDraftPolicyRequest, GetSandboxConfigRequest, GetSandboxProviderEnvironmentRequest,
IssueSandboxTokenRequest, NetworkActivitySummary, PolicyChunk, PolicySource, PolicyStatus,
RefreshSandboxTokenRequest, ReportEndpointStatusRequest, ReportPolicyStatusRequest,
SandboxPolicy as ProtoSandboxPolicy, SubmitPolicyAnalysisRequest, SubmitPolicyAnalysisResponse,
UpdateConfigRequest, open_shell_client::OpenShellClient, workspace_selector,
};
use crate::sandbox_env;
use miette::{IntoDiagnostic, Result, WrapErr};
@@ -1331,4 +1333,53 @@ impl CachedOpenShellClient {
Ok(())
}
/// Report the latest network results for all configured tool endpoints.
///
/// The shared snapshot contains only endpoint identifiers, typed results,
/// and the endpoints observed in the current batch, so this RPC cannot
/// accidentally attach request or credential material to public status.
pub async fn report_endpoint_status(
&self,
sandbox_id: &str,
snapshot: &EndpointStatusSnapshot,
) -> Result<()> {
let observations = snapshot
.endpoints
.iter()
.map(|endpoint| ProtoEndpointObservation {
endpoint_id: endpoint.endpoint_id.clone(),
result: match endpoint.result {
EndpointResult::NoObservedExchange => ProtoEndpointResult::NoObservedExchange,
EndpointResult::HttpResponseReceived => {
ProtoEndpointResult::HttpResponseReceived
}
EndpointResult::PolicyDenied => ProtoEndpointResult::PolicyDenied,
EndpointResult::CredentialUnavailable => {
ProtoEndpointResult::CredentialUnavailable
}
EndpointResult::TlsFailed => ProtoEndpointResult::TlsFailed,
EndpointResult::TransportFailed => ProtoEndpointResult::TransportFailed,
EndpointResult::UpstreamRejected => ProtoEndpointResult::UpstreamRejected,
}
.into(),
})
.collect();
self.client
.clone()
.report_endpoint_status(ReportEndpointStatusRequest {
sandbox_id: sandbox_id.to_string(),
policy_hash: snapshot.config_version.policy_hash.clone(),
provider_env_revision: snapshot.config_version.provider_env_revision,
observations,
observed_endpoint_ids: snapshot.observed_endpoint_ids.clone(),
supervisor_session_id: snapshot.supervisor_session_id.clone(),
report_sequence: snapshot.report_sequence,
})
.await
.into_diagnostic()?;
Ok(())
}
}
crates/openshell-core/src/lib.rs
+2
View File
@@ -18,6 +18,7 @@ pub mod driver_mounts;
pub mod driver_utils;
pub mod dynamic_string_allowlist;
pub mod endpoint_path;
pub mod endpoint_status;
pub mod error;
#[cfg(unix)]
pub mod external_driver_socket;
@@ -37,6 +38,7 @@ pub mod net;
pub mod oauth;
pub mod paths;
pub mod policy;
pub mod policy_identity;
pub mod progress;
pub mod proposals;
pub mod proto;
crates/openshell-core/src/mcp.rs
+11 -2
View File
@@ -1,7 +1,7 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
//! OpenShell-owned MCP protocol revisions and immutable batch-shape metadata.
//! `OpenShell`-owned MCP protocol revisions and immutable batch-shape metadata.
use std::collections::BTreeSet;
use std::fmt;
@@ -14,6 +14,15 @@ use crate::proto::{McpOptions, ProviderProfile};
/// The bound is distinct from the separately enforced request-body byte limit.
pub const MAX_MCP_LEGACY_BATCH_MESSAGES: usize = 64;
/// Return whether a policy protocol name denotes MCP.
///
/// Protocol names are case-insensitive throughout policy validation and
/// execution, so every MCP-specific projection must use the same predicate.
#[must_use]
pub fn is_mcp_protocol(protocol: &str) -> bool {
protocol.trim().eq_ignore_ascii_case("mcp")
}
/// Stable MCP protocol revisions accepted by `OpenShell` policy configuration.
///
/// Variant order is the canonical semantic order used when normalizing policy
@@ -46,7 +55,7 @@ pub const DEFAULT_MCP_PROTOCOL_VERSION: McpProtocolVersion = McpProtocolVersion:
/// original evidence. MCP-shaped data on non-MCP endpoints is also untouched.
pub fn normalize_provider_profile_mcp_fields(profile: &mut ProviderProfile) {
for endpoint in &mut profile.endpoints {
if !endpoint.protocol.eq_ignore_ascii_case("mcp") {
if !is_mcp_protocol(&endpoint.protocol) {
continue;
}
crates/openshell-core/src/policy_identity.rs
+177
View File
@@ -0,0 +1,177 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
//! Canonical policy encoding and deterministic policy hashes.
use crate::proto::{
NetworkMiddlewareConfig, NetworkPolicyRule, SandboxPolicy as ProtoSandboxPolicy,
};
use prost::Message;
use sha2::{Digest, Sha256};
use std::collections::HashMap;
#[allow(clippy::cast_possible_truncation)]
const fn canonical_size(value: usize) -> [u8; 8] {
// Every supported Rust pointer width fits in u64, so encoding a collection
// length or index needs no fallible conversion.
(value as u64).to_le_bytes()
}
fn append_canonical_bytes(out: &mut Vec<u8>, value: &[u8]) {
out.extend_from_slice(&canonical_size(value.len()));
out.extend_from_slice(value);
}
fn append_canonical_message<M: Message>(out: &mut Vec<u8>, value: &M) {
append_canonical_bytes(out, &value.encode_to_vec());
}
fn append_sorted_message_map<M: Message>(
out: &mut Vec<u8>,
label: &[u8],
values: &HashMap<String, M>,
) {
append_canonical_bytes(out, label);
let mut entries = values.iter().collect::<Vec<_>>();
entries.sort_by_key(|(key, _)| key.as_str());
out.extend_from_slice(&canonical_size(entries.len()));
for (key, value) in entries {
append_canonical_bytes(out, key.as_bytes());
append_canonical_message(out, value);
}
}
/// Encode a policy rule without depending on randomized protobuf map order.
pub fn canonical_rule_bytes(rule: &NetworkPolicyRule) -> Vec<u8> {
let mut map_free = rule.clone();
for endpoint in &mut map_free.endpoints {
endpoint.graphql_persisted_queries.clear();
for rule in &mut endpoint.rules {
if let Some(allow) = &mut rule.allow {
allow.query.clear();
allow.params.clear();
}
}
for deny in &mut endpoint.deny_rules {
deny.query.clear();
deny.params.clear();
}
}
let mut out = Vec::new();
append_canonical_message(&mut out, &map_free);
for (endpoint_index, endpoint) in rule.endpoints.iter().enumerate() {
out.extend_from_slice(&canonical_size(endpoint_index));
append_sorted_message_map(
&mut out,
b"graphql_persisted_queries",
&endpoint.graphql_persisted_queries,
);
for (rule_index, rule) in endpoint.rules.iter().enumerate() {
out.extend_from_slice(&canonical_size(rule_index));
if let Some(allow) = &rule.allow {
append_sorted_message_map(&mut out, b"allow_query", &allow.query);
append_sorted_message_map(&mut out, b"allow_params", &allow.params);
}
}
for (rule_index, deny) in endpoint.deny_rules.iter().enumerate() {
out.extend_from_slice(&canonical_size(rule_index));
append_sorted_message_map(&mut out, b"deny_query", &deny.query);
append_sorted_message_map(&mut out, b"deny_params", &deny.params);
}
}
out
}
fn canonical_struct_bytes(value: &prost_types::Struct) -> Vec<u8> {
let mut out = Vec::new();
let mut fields = value.fields.iter().collect::<Vec<_>>();
fields.sort_by_key(|(key, _)| key.as_str());
out.extend_from_slice(&canonical_size(fields.len()));
for (key, value) in fields {
append_canonical_bytes(&mut out, key.as_bytes());
append_canonical_bytes(&mut out, &canonical_value_bytes(value));
}
out
}
fn canonical_value_bytes(value: &prost_types::Value) -> Vec<u8> {
use prost_types::value::Kind;
let mut out = Vec::new();
match &value.kind {
None => out.push(0),
Some(Kind::NullValue(value)) => {
out.push(1);
out.extend_from_slice(&value.to_le_bytes());
}
Some(Kind::NumberValue(value)) => {
out.push(2);
out.extend_from_slice(&value.to_bits().to_le_bytes());
}
Some(Kind::StringValue(value)) => {
out.push(3);
append_canonical_bytes(&mut out, value.as_bytes());
}
Some(Kind::BoolValue(value)) => {
out.push(4);
out.push(u8::from(*value));
}
Some(Kind::StructValue(value)) => {
out.push(5);
append_canonical_bytes(&mut out, &canonical_struct_bytes(value));
}
Some(Kind::ListValue(value)) => {
out.push(6);
out.extend_from_slice(&canonical_size(value.values.len()));
for item in &value.values {
append_canonical_bytes(&mut out, &canonical_value_bytes(item));
}
}
}
out
}
fn canonical_middleware_bytes(middleware: &NetworkMiddlewareConfig) -> Vec<u8> {
let mut map_free = middleware.clone();
map_free.config = None;
let mut out = Vec::new();
append_canonical_message(&mut out, &map_free);
if let Some(config) = &middleware.config {
append_canonical_bytes(&mut out, &canonical_struct_bytes(config));
}
out
}
fn canonical_policy_bytes(policy: &ProtoSandboxPolicy) -> Vec<u8> {
let mut map_free = policy.clone();
map_free.network_policies.clear();
map_free.network_middlewares.clear();
let mut out = Vec::new();
append_canonical_message(&mut out, &map_free);
let mut policy_entries = policy.network_policies.iter().collect::<Vec<_>>();
policy_entries.sort_by_key(|(key, _)| key.as_str());
append_canonical_bytes(&mut out, b"network_policies");
out.extend_from_slice(&canonical_size(policy_entries.len()));
for (key, rule) in policy_entries {
append_canonical_bytes(&mut out, key.as_bytes());
append_canonical_bytes(&mut out, &canonical_rule_bytes(rule));
}
let mut middleware_entries = policy.network_middlewares.iter().collect::<Vec<_>>();
middleware_entries.sort_by_key(|(key, _)| key.as_str());
append_canonical_bytes(&mut out, b"network_middlewares");
out.extend_from_slice(&canonical_size(middleware_entries.len()));
for (key, middleware) in middleware_entries {
append_canonical_bytes(&mut out, key.as_bytes());
append_canonical_bytes(&mut out, &canonical_middleware_bytes(middleware));
}
out
}
/// Compute a deterministic SHA-256 hash of a `SandboxPolicy`, recursively
/// sorting every protobuf map while preserving repeated-field order.
pub fn deterministic_policy_hash(policy: &ProtoSandboxPolicy) -> String {
format!("{:x}", Sha256::digest(canonical_policy_bytes(policy)))
}
crates/openshell-driver-mxc/Cargo.toml
+3 -2
View File
@@ -30,9 +30,9 @@ tracing = { workspace = true }
thiserror = { workspace = true }
uuid = { workspace = true }
# ETW/TDH real-time consumer (Plane A audit). Windows-only so the Linux/WSL
# build stays an empty stub.
# ETW/TDH real-time consumer and host CONNECT proxy integration.
[target.'cfg(target_os = "windows")'.dependencies]
openshell-supervisor-network = { path = "../openshell-supervisor-network" }
windows = { workspace = true }
[dev-dependencies]
@@ -47,6 +47,7 @@ openshell-policy = { path = "../openshell-policy" }
# base64 and serde_json are already [dependencies] but dev-dependency resolution
# is independent; explicit entries make them visible to integration tests.
base64 = { workspace = true }
rustls = { workspace = true }
serde_json = { workspace = true }
# Used by the drift guard test (handled_fields_inventory) to parse YAML into a
# generic serde_json::Value for key enumeration.
crates/openshell-driver-mxc/README.md
+54 -15
View File
@@ -16,13 +16,17 @@ readiness; there is no in-sandbox supervisor or `ConnectSupervisor` relay.
## Capability Matrix
| Capability | MXC driver |
|---|---|
| Filesystem policy | Read-only/read-write grants come only from `SandboxPolicy`. `process_container` enforces default-deny; `isolation_session` is an explicit grant-only compatibility mode. |
| Network policy | Rejected synchronously during sandbox creation until an enforcing egress path is bound. |
| Process policy | Unsupported; MXC supplies OS isolation only. |
| Interactive exec/connect/forward | Unsupported; the configured workload runs in-driver. |
| Restart durability | Unsupported; the in-memory registry cannot recover live sessions. |
| Capability | MXC driver | Closing it requires |
|---|---|---|
| Filesystem policy (read-write / read-only grants) | ✅ provision-time AppContainer shares | — |
| Governed egress (CONNECT proxy + OPA + L7) | Available behind `egress_proxy` on `process_container`; the driver starts a per-sandbox host CONNECT proxy, generates HTTPS MITM trust material, and injects the CA bundle into the sandbox process env | Gateway event-bus wiring follow-on |
| Network policy | Split into MXC `network.proxy` + trimmed OpenShell policy on `process_container`; `isolation_session` still rejects network config | MXC feedback item M1 for persistent sessions |
| Network middleware | ❌ rejected before launch because the MXC host proxy does not receive the gateway middleware registry | Gateway middleware-registry injection |
| Process policy (seccomp, uid/gid) | ❌ host-side governance design; OS isolation only | not pursued |
| Interactive exec/connect/forward | ❌ exec runs in-driver, no client attach | gateway interactive-exec surgery (follow-on) |
| Bundled agent image | ❌ no OCI image; relies on Windows host install | — |
| Restart durability | ❌ in-memory registry; restart orphans live sessions | follow-on |
| Concurrent sandboxes | ⚠️ isolation_session v1 is single-session | MXC backend feature |
The filesystem enforcement proof has two paths:
@@ -41,10 +45,18 @@ backend = "process_container"
default_configuration_id = "composable"
pc_least_privilege = false
pc_capabilities = []
# Pattern-C governed egress. The address is a loopback seed; each sandbox
# receives a unique ephemeral proxy port.
egress_proxy = false
egress_proxy_addr = ""
debug = false
etw_audit = false
```
When `egress_proxy` is enabled, `egress_proxy_addr` must be a loopback
`IP:PORT` seed. The driver preserves the configured IP and allocates a unique
ephemeral port for each sandbox's `network.proxy` redirect.
Supply workload settings for each sandbox. The public config is keyed by driver name; the gateway forwards only the inner `mxc` object to the driver:
```powershell
@@ -55,9 +67,9 @@ openshell sandbox create --name mxc-demo --policy demo.yaml `
The `command` array is required and preserves Windows argument boundaries. `cwd` is optional. Environment variables come from the standard sandbox and template environment maps; the driver never copies values from the gateway host environment.
Network policy support depends on the selected containment path. Policy
replacement and merge updates use the gateway's standard sandbox configuration
revision contract.
The host CONNECT proxy enforces network policy when governed egress is enabled.
Live policy replacement or merge updates remain unsupported; delete and recreate
the sandbox to apply a different policy.
When `etw_audit` is enabled, each gateway process owns a distinct real-time ETW
session named from the stable `OpenShell-MXC-ETW` prefix, its process ID, and a
@@ -91,6 +103,9 @@ them. Records without matching generation evidence remain unattributed.
- Windows 11 Insider build ≥ 26300.8553
- `IsoSessionApp.dll` present and registered
- `wxc-exec.exe` built with `--features isolation_session`
- Any enforced App Control policy allows both `openshell-gateway.exe` and
`openshell.exe`. Diagnose executable blocks with event 3077 in the
`Microsoft-Windows-CodeIntegrity/Operational` log.
For off-box smoke tests against the in-process mock shim (no `wxc-exec`,
no isolation session needed), set `OPENSHELL_MXC_MOCK_WXC=1`.
@@ -103,9 +118,30 @@ invokes `wxc-exec`. Mapping failure therefore returns from `CreateSandbox`
without leaving a partial sandbox. There is no in-process policy side channel
or MXC-specific gateway composition variant.
`EmbeddedPolicyMapper` calls the embedded [`policy_map`](src/policy_map/) module directly and normalizes filesystem paths to Windows form. It does not add gateway-configured host paths. The policy supplied for the sandbox is the only source of filesystem grants.
When `egress_proxy` is enabled, `EmbeddedPolicyMapper` uses `split_policy`
instead: MXC receives filesystem grants plus a loopback `network.proxy`
redirect, and the driver starts a host CONNECT proxy from the trimmed
network-only `SandboxPolicy`. Policies containing `network_middlewares` are
rejected synchronously until this host-proxy path can receive the gateway's
built-in and remote middleware registry. The proxy uses the configured agent
command as the static sandbox process identity because MXC does not expose
Linux-style procfs socket ownership. For HTTPS L7 inspection, the host proxy generates a
per-sandbox CA and injects `NODE_EXTRA_CA_CERTS`, `DENO_CERT`, `SSL_CERT_FILE`,
`REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`, and `GIT_SSL_CAINFO` into the agent
process env. It does not add the generated CA directory to MXC read-only grants:
released `wxc-exec` BaseContainer builds require `WRITE_DAC` on every such
grant and reject the user-owned proxy temp directory. Instead, the driver adds
the sandbox-unique directory as an internal read-write share so HTTPS clients
can read the injected paths. The directory contains only public CA certificates;
the ephemeral CA private key remains in the host proxy's memory. The driver
seeds only `SYSTEMROOT`, `WINDIR`, `PATH`, `COMSPEC`, and `LOCALAPPDATA` from the
gateway host before applying sandbox and TLS overrides, so required Windows
bootstrap values remain available without exposing the gateway's full
environment. The development export surface remains the
[`policy-to-mxc`](examples/policy-to-mxc.rs) example; there is no production
`openshell policy export-mxc` subcommand yet.
The mapper retains an internal policy-splitting seam for future development, but the runtime exposes no governed-egress switch. Any network rule fails closed until an enforcing proxy is implemented and bound to the sandbox lifecycle.
If governed egress is disabled, any network rule fails closed rather than launching without an enforcement path.
Parity and matrix tests under [`tests/`](tests/) cover the mapper on the Windows MSVC lane. The driver performs this mapping automatically; there is no separate policy-export command or example.
@@ -126,7 +162,7 @@ exits 0 rather than failing.
| Task | What it runs | When to use |
|---|---|---|
| `windows:test:mxc-real:x64` | `tests/wxc_exec_real.rs` — Tier-2 invoker tests with `--ignored --test-threads=1` | Pre-merge on any Windows host that has `wxc-exec`; dry-run tests always pass; enforcement tests probe-gate themselves |
| `windows:test:mxc-real:x64` | `tests/wxc_exec_real.rs` — Tier-2 invoker tests with `--ignored --test-threads=1`, including an HTTPS request through the host proxy | Pre-merge on any Windows host that has `wxc-exec`; dry-run tests always pass; enforcement tests probe-gate themselves |
| `windows:e2e:mxc` | `examples/run-mxc-e2e.ps1` — Tier-3 scenario runner, real binary, probe-gated | Demo box / nightly; needs the gateway + CLI binaries in the script directory |
| `windows:e2e:mxc:mock` | Same runner with `-Mock` — wiring-only, no real `wxc-exec` needed | Any Windows host (CI, dev machine); validates wiring and the network-reject scenario |
@@ -135,6 +171,10 @@ exits 0 rather than failing.
and a `verdicts` object). Run it before the real-MXC lane to understand what
will PASS vs SKIP on a given host:
The probe uses a unique, user-owned Windows temp directory for every run.
MXC treats config paths literally (it does not expand `%TEMP%`), and the
per-run directory keeps AppContainer+DACL fallback mutations narrowly scoped.
```powershell
powershell -NoProfile -ExecutionPolicy Bypass `
-File crates/openshell-driver-mxc/examples/probe-mxc-host.ps1
@@ -149,7 +189,6 @@ velocity keys not enabled, isolation_session absent).
## Deferred work
- **Interactive exec/connect/forward** → `adapt-openshell-gateway-windows`
- **Governed egress** remains fail-closed until an enforcing proxy is implemented and bound to sandbox lifecycle.
- **Interactive exec/connect/forward** — gateway interactive-exec surgery (follow-on)
- **Restart durability** (deprovision orphaned sessions on startup) → follow-on
- **GPU passthrough** → not pursued in host-side-governance design
crates/openshell-driver-mxc/examples/README-ocsf-audit.txt
+8 -2
View File
@@ -16,7 +16,7 @@ WHAT THIS PROVES / PRODUCES
OCSF classes you will see:
[6002] Application Lifecycle - sandbox created
[5019] Device Config State Change - OS policy / hardening / console
[5019] Device Config State Change - OS policy / hardening / proxy / console
[1007] Process Activity - in-sandbox process launch (+ executable identity)
[2004] Detection Finding - MXC setup activity errors (informational)
@@ -46,7 +46,7 @@ WHAT YOU GET BACK
FILES IN THIS PACKAGE
openshell-gateway.exe the gateway (self-contained; needs only VC++ runtime)
openshell.exe the CLI
mxc-ocsf-audit.toml gateway/driver config (process_container, etw_audit=true)
mxc-ocsf-audit.toml gateway/driver config (process_container, etw_audit=true, egress proxy)
ocsf-audit.yaml sandbox policy (read-write grant to the share dir)
run-ocsf-audit.ps1 the orchestrator you run
README-ocsf-audit.txt this file
@@ -54,6 +54,9 @@ FILES IN THIS PACKAGE
USEFUL OPTIONS
-SandboxCount <n> Create n sandboxes (default 2). More sandboxes = more events.
-NoProxy Skip the per-sandbox egress proxy. This omits ONLY the
SandboxProxyConfigured config event; everything else is
still produced. (Default is proxy ON for the full set.)
-ShareDir <path> Host folder granted read-write to the workload. The script
derives a disposable policy and per-sandbox config for it.
-WxcExecPath <path> Path to wxc-exec.exe on this box.
@@ -66,6 +69,9 @@ NOTES
that is unrelated to the OCSF audit path this example exercises.
- A "supervisor session not connected" / ssh 255 message during sandbox create
is EXPECTED on MXC and harmless - the agent already ran in-driver.
- The proxy path requires the host-side CONNECT proxy and an absolute agent
binary (the packaged config uses C:\Windows\System32\cmd.exe); the run script
handles this for you.
- The Sandboxing provider reports the sandbox entry-point process, not the full
in-sandbox process tree. Deep process-tree auditing would need a second ETW
source (Microsoft-Windows-Kernel-Process) and is out of scope for this trail.
crates/openshell-driver-mxc/examples/mxc-ocsf-audit.toml
+13 -3
View File
@@ -9,9 +9,9 @@
# Detection Finding [2004] — written to a durable JSONL log, just like the Linux
# OCSF pipeline.
#
# run-ocsf-audit.ps1 patches wxc_exec_path, backend, and etw_audit into a
# disposable copy of this file. Workload command and cwd are sandbox-scoped and
# passed separately through --driver-config-json.
# run-ocsf-audit.ps1 patches wxc_exec_path, backend, etw_audit and the
# egress-proxy switch into a disposable copy of this file. Workload command and
# cwd are sandbox-scoped and passed separately through --driver-config-json.
[openshell.drivers.mxc]
# Path to wxc-exec.exe on the box (patched by the run script; default is the
@@ -28,3 +28,13 @@ debug = false
# Turn ON the Plane-A ETW -> OCSF audit consumer. This is the core of the example.
etw_audit = true
# Per-sandbox governed egress. Enabling this makes the driver start a host CONNECT
# proxy and hand MXC a `network.proxy` redirect, which is what makes MXC emit the
# SandboxProxyConfigured event — the config event mapped to OCSF CONFIG [5019]
# that completes full event coverage. Requires backend = process_container and a
# loopback (127.0.0.1) seed address; the driver allocates a unique ephemeral port
# per sandbox from this seed. Run-ocsf-audit.ps1 disables this when passed
# -NoProxy.
egress_proxy = true
egress_proxy_addr = "127.0.0.1:18080"
crates/openshell-driver-mxc/examples/ocsf-audit.yaml
+4
View File
@@ -6,6 +6,10 @@
# Minimal filesystem policy granting the workload folder read-write; everything
# else is default-deny. run-ocsf-audit.ps1 copies this policy into the result
# bundle and replaces the default grant with -ShareDir for that run.
#
# No network_policies block is needed here: the per-sandbox egress proxy is driven
# by `egress_proxy = true` in mxc-ocsf-audit.toml (that is what makes MXC emit the
# SandboxProxyConfigured event we map to OCSF), not by a policy rule.
version: 1
filesystem_policy:
crates/openshell-driver-mxc/examples/probe-mxc-host.ps1
+19 -5
View File
@@ -119,6 +119,10 @@ $pcTrialMessage = "wxc-exec not found"
$isoTrialResult = "absent"
$isoTrialMessage = "wxc-exec not found"
$probeTempPath = Join-Path ([System.IO.Path]::GetTempPath()) ("openshell-mxc-probe-" + [Guid]::NewGuid().ToString("N"))
New-Item -ItemType Directory -Path $probeTempPath | Out-Null
try {
if ($wxcInfo.exists) {
# --probe
$probeResult = Invoke-WxcProbe -wxc $WxcExecPath
@@ -131,11 +135,11 @@ if ($wxcInfo.exists) {
containment = "processcontainer"
process = @{
commandLine = "cmd /c exit 0"
cwd = "%TEMP%"
cwd = $probeTempPath
timeout = 0
}
filesystem = @{
readwritePaths = @("%TEMP%")
readwritePaths = @($probeTempPath)
}
}
$dryResult = Invoke-WxcDryRun -wxc $WxcExecPath -config $dryConfig
@@ -149,15 +153,22 @@ if ($wxcInfo.exists) {
containment = "processcontainer"
process = @{
commandLine = "cmd /c exit 0"
cwd = "%TEMP%"
timeout = 10
cwd = $probeTempPath
timeout = 30000
}
filesystem = @{
readwritePaths = @("%TEMP%")
readwritePaths = @($probeTempPath)
}
processContainer = @{
leastPrivilege = $false
}
# cmd.exe needs the Win32k calls represented by disable = false.
# Filesystem and network restrictions remain default-deny.
ui = @{
disable = $false
clipboard = "none"
injection = $false
}
}
$pcResult = Invoke-WxcPhase -wxc $WxcExecPath -config $pcConfig
$pcOutput = $pcResult.Output
@@ -260,6 +271,9 @@ if ($wxcInfo.exists) {
$isoTrialMessage = "exit $($isoResult.ExitCode): $isoOutput"
}
}
} finally {
Remove-Item -LiteralPath $probeTempPath -Recurse -Force -ErrorAction SilentlyContinue
}
# ── Verdicts ──────────────────────────────────────────────────────────────────
crates/openshell-driver-mxc/examples/run-mxc-e2e.ps1
+573 -313
View File
File diff suppressed because it is too large Load Diff
crates/openshell-driver-mxc/examples/run-ocsf-audit.ps1
+23 -3
View File
@@ -23,6 +23,9 @@
# powershell -NoProfile -ExecutionPolicy Bypass -File .\run-ocsf-audit.ps1 `
# -WxcExecPath C:\mxc-kit\bin\wxc-exec.exe
#
# By default the per-sandbox egress proxy is ON so the full event set (including
# SandboxProxyConfigured) is produced. Pass -NoProxy to omit only that one event.
#
# The deliverable is the OCSF audit log (openshell-ocsf.<date>.log) inside the
# results-*.zip the script produces. Pass -ShareOut '\\server\share' to also copy
# the bundle to a shared location (off by default).
@@ -35,6 +38,8 @@ param(
[string] $ShareDir = "C:\work\openshell-mxc-demo",
# How many sandboxes to create (each drives a full event burst).
[int] $SandboxCount = 2,
# Disable the per-sandbox egress proxy (omits the SandboxProxyConfigured event).
[switch] $NoProxy,
# Gateway bind port (matches the gateway default) + CLI registration name.
[int] $Port = 17670,
[string] $GatewayName = "openshell-mxc-ocsf",
@@ -90,6 +95,7 @@ $helloPath = Join-Path $ShareDir "hello.txt"
$gw = $null
$gatewayEtwSessions = @()
$passed = $true
$proxyOn = -not $NoProxy
try {
# 1. Validate artifacts + privilege.
@@ -127,6 +133,12 @@ try {
} else {
$tomlText = [regex]::Replace($tomlText, '(?m)^\[openshell\.drivers\.mxc\]\s*$', "[openshell.drivers.mxc]`r`netw_audit = true")
}
$proxyVal = if ($proxyOn) { 'true' } else { 'false' }
if ($tomlText -match '(?m)^\s*#?\s*egress_proxy\s*=') {
$tomlText = [regex]::Replace($tomlText, '(?m)^\s*#?\s*egress_proxy\s*=.*$', "egress_proxy = $proxyVal")
} else {
$tomlText = [regex]::Replace($tomlText, '(?m)^\[openshell\.drivers\.mxc\]\s*$', "[openshell.drivers.mxc]`r`negress_proxy = $proxyVal")
}
Set-Content $toml -Value $tomlText -Encoding UTF8
$shareDirPolicy = $ShareDir.Replace('\', '/')
@@ -156,7 +168,7 @@ try {
$driverConfig
}
Info "backend=process_container etw_audit=true"
Info "backend=process_container etw_audit=true egress_proxy=$proxyVal"
Info "workload cwd=$shareDirPolicy policy grant=$shareDirPolicy"
# 3. Port must be free. Auto-clear a stale OUR-gateway; refuse anything else.
@@ -199,6 +211,10 @@ try {
$env:OPENSHELL_WXC_EXEC_PATH = $WxcExecPath
$env:OPENSHELL_OCSF_JSON = "1"
$env:OPENSHELL_OCSF_LOG_DIR = $resultDir
# Config path goes through the env var (clap: OPENSHELL_GATEWAY_CONFIG), NOT a
# --config token: Start-Process -ArgumentList does not quote array elements, so a
# config path containing a space gets split and the gateway's arg parser rejects it.
$env:OPENSHELL_GATEWAY_CONFIG = $toml
Remove-Item Env:OPENSHELL_MXC_MOCK_WXC -ErrorAction SilentlyContinue
# 7. Start the gateway (background, TLS disabled on the loopback control plane).
@@ -206,7 +222,7 @@ try {
$gwLog = Join-Path $resultDir "gateway.log"
$gwErrLog = Join-Path $resultDir "gateway.err.log"
$gw = Start-Process -FilePath $gateway `
-ArgumentList @("--disable-tls", "--config", $toml, "--port", $Port, "--log-level", "info") `
-ArgumentList @("--disable-tls", "--port", $Port, "--log-level", "info") `
-WorkingDirectory $here -PassThru -NoNewWindow `
-RedirectStandardOutput $gwLog -RedirectStandardError $gwErrLog
Info "gateway pid $($gw.Id); logs -> $(Split-Path $gwLog -Leaf) (+ .err)"
@@ -314,7 +330,9 @@ finally {
# Event-type coverage (detected from the human-readable shorthand lines).
function Seen([string]$pat) { [bool]($logText | Select-String -Pattern $pat -Quiet) }
# Expected happy-path ETW->OCSF event types for this run.
# Expected happy-path ETW->OCSF event types for THIS run. The egress-proxy
# event only fires when the proxy is enabled, so it only counts toward the
# expected total when -NoProxy was NOT passed.
$coreEvents = [ordered]@{
"sandbox lifecycle (start)" = Seen "(?i)ocsf:.*LIFECYCLE:"
"OS policy enforced" = Seen "(?i)ocsf:.*OS policy enforced"
@@ -324,6 +342,7 @@ finally {
"console reference plumbed" = Seen "(?i)ocsf:.*console reference plumbed"
"process launch (executable identity)" = Seen "(?i)ocsf:.*PROC:LAUNCH"
}
if ($proxyOn) { $coreEvents["egress proxy configured"] = Seen "(?i)ocsf:.*proxy configured" }
# Findings are anomaly / fallback signals - reported separately, NOT part of
# the expected-coverage denominator (a clean run may emit none).
@@ -354,6 +373,7 @@ user : $env:USERNAME (admin=$admin perfLogUsers=$plu)
verdict : $verdict
event coverage : $coreObserved of $coreExpected expected event types fired (+ $findingsObserved anomaly finding(s))
queue overload : $(if ($consumerOverloaded) { 'YES - ETW records dropped; audit coverage gap' } else { 'no dropped ETW records observed' })
proxy : $(if ($proxyOn) { 'on (full event set)' } else { 'off (-NoProxy; omits egress proxy event)' })
workload output : $(if ($workloadCompleted) { $helloPath } else { '(missing)' })
wxc_exec : $WxcExecPath
backend : process_container
crates/openshell-driver-mxc/src/driver.rs
+451 -26
View File
@@ -4,7 +4,7 @@
//! MXC compute backend: lifecycle logic, in-memory registry, exec-in-driver,
//! and self-reported readiness.
use crate::mxc::{MxcFilesystem, MxcProcess, MxcProcessContainer, WxcExecInvoker};
use crate::mxc::{MxcFilesystem, MxcNetwork, MxcProcess, MxcProcessContainer, WxcExecInvoker};
use crate::policy::{EmbeddedPolicyMapper, MapCtx, MappedConfig, PolicyMapper};
use futures::Stream;
use openshell_core::gpu::{driver_gpu_requirements, effective_driver_gpu_count};
@@ -17,6 +17,8 @@ use openshell_core::proto::compute::v1::{
use openshell_core::proto_struct::struct_to_json_value;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::net::SocketAddr;
use std::path::PathBuf;
use std::pin::Pin;
use std::sync::Arc;
use tokio::sync::{Mutex, broadcast, mpsc, watch};
@@ -65,6 +67,14 @@ pub struct MxcComputeConfig {
/// MXC `configurationId` for isolation session. Default: `"composable"`.
/// Never use `"small"` (known OS bug).
pub default_configuration_id: String,
/// Enable Pattern-C governed egress. When true, MXC receives filesystem
/// grants plus a `network.proxy` redirect and the host CONNECT proxy
/// receives the trimmed network-only policy.
pub egress_proxy: bool,
/// Loopback `IP:PORT` seed for MXC `network.proxy` while governed egress is
/// enabled. The driver preserves the loopback IP and allocates a unique
/// ephemeral port per sandbox.
pub egress_proxy_addr: String,
/// Enable `--debug` flag on `wxc-exec` invocations.
pub debug: bool,
@@ -82,6 +92,8 @@ impl Default for MxcComputeConfig {
pc_least_privilege: false,
pc_capabilities: Vec::new(),
default_configuration_id: crate::mxc::DEFAULT_CONFIGURATION_ID.into(),
egress_proxy: false,
egress_proxy_addr: String::new(),
debug: false,
etw_audit: false,
@@ -118,6 +130,9 @@ struct SandboxEntry {
lifecycle_gate: Arc<Mutex<()>>,
monitor_cancel: Option<watch::Sender<bool>>,
monitor_task: Option<JoinHandle<()>>,
trimmed_policy: Option<SandboxPolicy>,
proxy_addr: Option<SocketAddr>,
host_proxy: Option<openshell_supervisor_network::host::HostProxyHandle>,
}
impl std::fmt::Debug for SandboxEntry {
@@ -226,15 +241,30 @@ fn sandbox_config(sandbox: &DriverSandbox) -> Result<MxcSandboxConfig, tonic::St
Ok(config)
}
// Minimum non-secret Windows environment needed by CreateProcessW and the
// AppContainer DACL fallback before the workload runtime starts.
const MINIMAL_WINDOWS_BOOTSTRAP_ENV: [&str; 5] =
["SYSTEMROOT", "WINDIR", "PATH", "COMSPEC", "LOCALAPPDATA"];
fn sandbox_environment(sandbox: &DriverSandbox) -> Vec<String> {
let Some(spec) = sandbox.spec.as_ref() else {
return Vec::new();
};
let mut environment = spec
.template
.as_ref()
.map_or_else(HashMap::new, |template| template.environment.clone());
environment.extend(spec.environment.clone());
// Released wxc-exec ProcessContainer builds start from the explicit
// process environment. Seed only the non-secret Windows bootstrap values;
// copying the gateway's full environment would leak unrelated host secrets
// into untrusted sandbox workloads.
let mut environment = MINIMAL_WINDOWS_BOOTSTRAP_ENV
.iter()
.filter_map(|key| {
std::env::var(key)
.ok()
.map(|value| ((*key).to_string(), value))
})
.collect::<HashMap<_, _>>();
if let Some(spec) = sandbox.spec.as_ref() {
if let Some(template) = spec.template.as_ref() {
environment.extend(template.environment.clone());
}
environment.extend(spec.environment.clone());
}
let mut environment = environment
.into_iter()
.map(|(key, value)| format!("{key}={value}"))
@@ -243,6 +273,43 @@ fn sandbox_environment(sandbox: &DriverSandbox) -> Vec<String> {
environment
}
fn configured_egress_addr(config: &MxcComputeConfig) -> Result<Option<SocketAddr>, tonic::Status> {
if !config.egress_proxy {
return Ok(None);
}
if config.backend == MxcBackend::IsolationSession {
return Err(tonic::Status::invalid_argument(
"mxc governed egress requires process_container; network.proxy is not supported on isolation_session until MXC M1 lands",
));
}
let raw = config.egress_proxy_addr.trim();
if raw.is_empty() {
return Err(tonic::Status::invalid_argument(
"mxc egress_proxy_addr is required when egress_proxy is enabled",
));
}
let addr = raw.parse::<SocketAddr>().map_err(|error| {
tonic::Status::invalid_argument(format!(
"mxc egress_proxy_addr must be an IP:PORT socket address: {error}"
))
})?;
if addr.ip() != std::net::IpAddr::from([127, 0, 0, 1]) {
return Err(tonic::Status::invalid_argument(format!(
"mxc egress_proxy_addr must be 127.0.0.1:PORT because MXC 0.6.0-alpha can encode only a localhost proxy port (got {})",
addr.ip()
)));
}
Ok(Some(addr))
}
fn allocate_sandbox_proxy_addr(
configured: SocketAddr,
) -> std::io::Result<(SocketAddr, std::net::TcpListener)> {
let reservation = std::net::TcpListener::bind(SocketAddr::new(configured.ip(), 0))?;
let addr = reservation.local_addr()?;
Ok((addr, reservation))
}
fn encode_windows_command_line(args: &[String]) -> String {
args.iter()
.map(|arg| quote_windows_argument(arg))
@@ -276,6 +343,66 @@ fn quote_windows_argument(arg: &str) -> String {
quoted.push('"');
quoted
}
fn host_proxy_binary_path(config: &MxcSandboxConfig) -> PathBuf {
config
.command
.first()
.filter(|command| !command.trim().is_empty())
.map_or_else(|| PathBuf::from("mxc-agent"), PathBuf::from)
}
const TLS_ENV_KEYS: [&str; 6] = [
"NODE_EXTRA_CA_CERTS",
"DENO_CERT",
"SSL_CERT_FILE",
"REQUESTS_CA_BUNDLE",
"CURL_CA_BUNDLE",
"GIT_SSL_CAINFO",
];
fn append_tls_env_vars(env: &mut Vec<String>, ca_paths: Option<&(PathBuf, PathBuf)>) {
let Some((ca_cert_path, combined_bundle_path)) = ca_paths else {
return;
};
env.retain(|entry| {
let key = entry.split_once('=').map_or(entry.as_str(), |(key, _)| key);
!TLS_ENV_KEYS
.iter()
.any(|candidate| key.eq_ignore_ascii_case(candidate))
});
let ca_cert_path = ca_cert_path.display().to_string();
let combined_bundle_path = combined_bundle_path.display().to_string();
env.extend([
format!("NODE_EXTRA_CA_CERTS={ca_cert_path}"),
format!("DENO_CERT={ca_cert_path}"),
format!("SSL_CERT_FILE={combined_bundle_path}"),
format!("REQUESTS_CA_BUNDLE={combined_bundle_path}"),
format!("CURL_CA_BUNDLE={combined_bundle_path}"),
format!("GIT_SSL_CAINFO={combined_bundle_path}"),
]);
}
fn append_tls_readwrite_grant(
readwrite_paths: &mut Vec<String>,
ca_paths: Option<&(PathBuf, PathBuf)>,
) {
let Some((ca_cert_path, _)) = ca_paths else {
return;
};
let Some(dir) = ca_cert_path.parent() else {
return;
};
let dir = dir.display().to_string();
if !readwrite_paths
.iter()
.any(|existing| existing.eq_ignore_ascii_case(&dir))
{
readwrite_paths.push(dir);
}
}
impl MxcComputeBackend {
pub fn new(config: MxcComputeConfig) -> Self {
let invoker = WxcExecInvoker::new(&config.wxc_exec_path, config.debug);
@@ -363,13 +490,14 @@ impl MxcComputeBackend {
&self,
sandbox_id: &str,
policy: Option<&SandboxPolicy>,
egress: Option<SocketAddr>,
) -> Result<MappedConfig, tonic::Status> {
self.policy_mapper
.map(
policy,
&MapCtx {
sandbox_id: sandbox_id.to_string(),
egress: None,
egress,
},
)
.map_err(|error| tonic::Status::invalid_argument(error.to_string()))
@@ -378,7 +506,8 @@ impl MxcComputeBackend {
pub fn validate_sandbox_create(&self, sandbox: &DriverSandbox) -> Result<(), tonic::Status> {
Self::validate_sandbox_fields(sandbox)?;
let policy = sandbox.spec.as_ref().and_then(|spec| spec.policy.as_ref());
self.map_sandbox_policy(&sandbox.id, policy)?;
let egress_addr = configured_egress_addr(&self.config)?;
self.map_sandbox_policy(&sandbox.id, policy, egress_addr)?;
Ok(())
}
pub async fn get_sandbox(&self, sandbox_name: &str) -> Option<DriverSandbox> {
@@ -399,12 +528,25 @@ impl MxcComputeBackend {
Self::validate_sandbox_fields(sandbox)?;
let sandbox_config = sandbox_config(sandbox)?;
let (egress_addr, reserved_proxy_listener) = match configured_egress_addr(&self.config)? {
Some(configured_addr) => {
let (addr, reservation) = allocate_sandbox_proxy_addr(configured_addr).map_err(
|error| {
tonic::Status::internal(format!(
"failed to allocate sandbox-unique MXC host egress proxy address from {configured_addr}: {error}"
))
},
)?;
(Some(addr), Some(reservation))
}
None => (None, None),
};
// Policy translation is deterministic and side-effect free. Do it before
// inserting the registry entry or launching MXC so invalid requests fail
// synchronously at the CreateSandbox boundary.
let policy = sandbox.spec.as_ref().and_then(|spec| spec.policy.as_ref());
let mapped = self.map_sandbox_policy(&sandbox_id, policy)?;
let mapped = self.map_sandbox_policy(&sandbox_id, policy, egress_addr)?;
if sandbox
.spec
@@ -452,6 +594,9 @@ impl MxcComputeBackend {
lifecycle_gate,
monitor_cancel: None,
monitor_task: None,
trimmed_policy: None,
proxy_addr: None,
host_proxy: None,
},
);
}
@@ -472,6 +617,7 @@ impl MxcComputeBackend {
sandbox,
sandbox_config,
mapped,
reserved_proxy_listener,
startup_guard,
)
.await;
@@ -524,6 +670,7 @@ impl MxcComputeBackend {
let mut registry = self.registry.lock().await;
if let Some(entry) = registry.get_mut(&sandbox_id) {
entry.isolation_stopped = isolation_stopped;
entry.host_proxy = None;
entry.phase_state = PhaseState::Stopped;
entry.sandbox = make_sandbox_with_condition(
&entry.sandbox,
@@ -668,29 +815,102 @@ async fn run_lifecycle(
sandbox: DriverSandbox,
sandbox_config: MxcSandboxConfig,
mapped: MappedConfig,
mut reserved_proxy_listener: Option<std::net::TcpListener>,
_startup_guard: tokio::sync::OwnedMutexGuard<()>,
) {
let sandbox_id = sandbox.id.clone();
let sandbox_name = sandbox.name.clone();
let trimmed_policy = mapped.trimmed_policy.clone();
let proxy_addr = mapped.proxy_addr;
let host_proxy = if !invoker.is_mock()
&& let (Some(addr), Some(proxy_policy)) = (proxy_addr, trimmed_policy.clone())
{
drop(reserved_proxy_listener.take());
match openshell_supervisor_network::host::start_host_proxy(
openshell_supervisor_network::host::HostProxyConfig {
bind_addr: addr,
policy: proxy_policy,
binary_path: host_proxy_binary_path(&sandbox_config),
sandbox_id: Some(sandbox_id.clone()),
sandbox_name: Some(sandbox_name.clone()),
openshell_endpoint: None,
provider_credentials: None,
agent_proposals: openshell_core::proposals::AgentProposals::default(),
denial_tx: None,
activity_tx: None,
},
)
.await
{
Ok(handle) => Some(handle),
Err(error) => {
set_failed(
&registry,
&watch_tx,
&sandbox,
&sandbox_id,
&format!("failed to start MXC host egress proxy at {addr}: {error}"),
)
.await;
return;
}
}
} else {
None
};
let host_proxy_ca_paths = host_proxy.as_ref().and_then(|proxy| proxy.ca_file_paths());
drop(reserved_proxy_listener.take());
if let Some(addr) = proxy_addr {
{
let mut registry = registry.lock().await;
if let Some(entry) = registry.get_mut(&sandbox_id) {
entry.trimmed_policy = trimmed_policy;
entry.proxy_addr = Some(addr);
entry.host_proxy = host_proxy;
}
}
let _ = watch_tx.send(platform_event(
sandbox_id.clone(),
"EgressRedirect",
format!("MXC egress redirected to OpenShell host CONNECT proxy at {addr}"),
));
}
// Released wxc-exec BaseContainer builds cannot provision the generated
// TLS directory as a read-only share because that path fails its WRITE_DAC
// setup. Grant the sandbox-unique directory read-write instead so the
// AppContainer can actually read the injected trust paths. The directory
// contains only public CA certificates; the CA private key remains in the
// host proxy's in-memory TLS state.
let mut readwrite_paths = mapped.readwrite_paths;
append_tls_readwrite_grant(&mut readwrite_paths, host_proxy_ca_paths.as_ref());
let readonly_paths = mapped.readonly_paths;
let filesystem = MxcFilesystem {
readwrite_paths: mapped.readwrite_paths,
readonly_paths: mapped.readonly_paths,
readwrite_paths,
readonly_paths,
// OpenShell's policy model has no explicit deny field; default-deny is
// implicit and enforced by processContainer at the OS boundary.
denied_paths: Vec::new(),
};
let command_line = encode_windows_command_line(&sandbox_config.command);
let mut environment = sandbox_environment(&sandbox);
append_tls_env_vars(&mut environment, host_proxy_ca_paths.as_ref());
info!(sandbox = %sandbox_name, count = environment.len(), "MXC process env vars");
let process = MxcProcess {
command_line: command_line.clone(),
cwd: sandbox_config.cwd,
env: sandbox_environment(&sandbox),
env: environment,
timeout: 0,
};
let network = proxy_addr.map(|addr| MxcNetwork {
default_policy: "block".into(),
proxy: Some(addr),
});
let child = match config.backend {
MxcBackend::IsolationSession => {
let iso_sandbox_id = match invoker
.provision(&config.default_configuration_id, filesystem, None)
.provision(&config.default_configuration_id, filesystem, network)
.await
{
Ok(id) => id,
@@ -748,7 +968,7 @@ async fn run_lifecycle(
capabilities: config.pc_capabilities.clone(),
};
match invoker
.run_oneshot(&sandbox_id, filesystem, process_container, process, None)
.run_oneshot(&sandbox_id, filesystem, process_container, process, network)
.await
{
Ok(child) => child,
@@ -890,6 +1110,7 @@ async fn monitor_exec(
);
let mut registry = registry.lock().await;
if let Some(entry) = registry.get_mut(&sandbox_id) {
entry.host_proxy = None;
entry.sandbox = done.clone();
entry.phase_state = PhaseState::Running;
}
@@ -917,6 +1138,7 @@ async fn monitor_exec(
);
let mut registry = registry.lock().await;
if let Some(entry) = registry.get_mut(&sandbox_id) {
entry.host_proxy = None;
entry.sandbox = failed.clone();
entry.phase_state = PhaseState::Failed(format!("exit code {code}"));
}
@@ -949,6 +1171,7 @@ async fn set_failed(
);
let mut reg = registry.lock().await;
if let Some(entry) = reg.get_mut(sandbox_id) {
entry.host_proxy = None;
entry.sandbox = failed.clone();
entry.phase_state = PhaseState::Failed(message.to_string());
}
@@ -992,7 +1215,9 @@ mod lifecycle_tests {
use super::*;
use futures::StreamExt;
use openshell_core::proto::compute::v1::{DriverSandboxSpec, DriverSandboxTemplate};
use openshell_core::proto::{FilesystemPolicy, SandboxPolicy};
use openshell_core::proto::{
FilesystemPolicy, MiddlewareEndpointSelector, NetworkMiddlewareConfig, SandboxPolicy,
};
use std::time::Duration;
fn driver_sandbox(id: &str) -> DriverSandbox {
@@ -1071,14 +1296,22 @@ mod lifecycle_tests {
#[test]
fn mxc_config_defaults_to_default_deny_process_container() {
assert_eq!(
MxcComputeConfig::default().backend,
MxcBackend::ProcessContainer
);
let config = MxcComputeConfig::default();
assert_eq!(config.backend, MxcBackend::ProcessContainer);
assert!(!config.egress_proxy);
assert!(config.egress_proxy_addr.is_empty());
}
#[test]
fn sandbox_environment_uses_sandbox_scope_with_spec_precedence() {
fn sandbox_proxy_addr_uses_ephemeral_loopback_port() {
let configured = "127.0.0.1:18080".parse().unwrap();
let (addr, _reservation) = allocate_sandbox_proxy_addr(configured).unwrap();
assert_eq!(addr.ip(), configured.ip());
assert_ne!(addr.port(), 0);
}
#[test]
fn sandbox_environment_inherits_host_with_spec_precedence() {
let mut sandbox = driver_sandbox("sb-env");
let spec = sandbox.spec.as_mut().unwrap();
spec.template
@@ -1088,10 +1321,63 @@ mod lifecycle_tests {
.insert("SHARED".into(), "template".into());
spec.environment.insert("SHARED".into(), "spec".into());
spec.environment.insert("TOKEN".into(), "value".into());
assert_eq!(
sandbox_environment(&sandbox),
vec!["SHARED=spec".to_string(), "TOKEN=value".to_string()]
let environment = sandbox_environment(&sandbox);
assert!(environment.contains(&"SHARED=spec".to_string()));
assert!(environment.contains(&"TOKEN=value".to_string()));
for key in MINIMAL_WINDOWS_BOOTSTRAP_ENV {
if let Ok(value) = std::env::var(key) {
assert!(environment.contains(&format!("{key}={value}")));
}
}
assert!(environment.iter().all(|entry| {
let key = entry.split_once('=').map_or(entry.as_str(), |(key, _)| key);
key == "SHARED" || key == "TOKEN" || MINIMAL_WINDOWS_BOOTSTRAP_ENV.contains(&key)
}));
}
#[test]
fn tls_env_vars_replace_user_trust_overrides() {
let tls_dir = std::env::temp_dir().join("openshell-mxc-tls-test");
let ca_cert = tls_dir.join("openshell-ca.pem");
let bundle = tls_dir.join("ca-bundle.pem");
let ca_cert_path = ca_cert.display().to_string();
let bundle_path = bundle.display().to_string();
let mut env = vec![
"FOO=bar".to_string(),
"SSL_CERT_FILE=C:\\old\\bundle.pem".to_string(),
"node_extra_ca_certs=C:\\old\\ca.pem".to_string(),
];
append_tls_env_vars(&mut env, Some(&(ca_cert, bundle)));
assert!(env.contains(&"FOO=bar".to_string()));
assert!(
!env.iter()
.any(|entry| entry == "SSL_CERT_FILE=C:\\old\\bundle.pem")
);
assert!(
!env.iter()
.any(|entry| entry == "node_extra_ca_certs=C:\\old\\ca.pem")
);
assert!(env.contains(&format!("NODE_EXTRA_CA_CERTS={ca_cert_path}")));
assert!(env.contains(&format!("DENO_CERT={ca_cert_path}")));
assert!(env.contains(&format!("SSL_CERT_FILE={bundle_path}")));
assert!(env.contains(&format!("REQUESTS_CA_BUNDLE={bundle_path}")));
assert!(env.contains(&format!("CURL_CA_BUNDLE={bundle_path}")));
assert!(env.contains(&format!("GIT_SSL_CAINFO={bundle_path}")));
}
#[test]
fn tls_readwrite_grant_adds_ca_directory_once() {
let tls_dir = std::env::temp_dir().join("openshell-mxc-tls-test");
let ca_cert = tls_dir.join("openshell-ca.pem");
let bundle = tls_dir.join("ca-bundle.pem");
let existing = tls_dir.display().to_string().to_ascii_lowercase();
let mut readwrite = vec![existing.clone()];
append_tls_readwrite_grant(&mut readwrite, Some(&(ca_cert, bundle)));
assert_eq!(readwrite, vec![existing]);
}
#[test]
@@ -1220,6 +1506,116 @@ mod lifecycle_tests {
);
}
#[tokio::test]
async fn split_path_provisions_with_proxy_redirect() {
use openshell_core::proto::{NetworkBinary, NetworkEndpoint, NetworkPolicyRule};
let tmp = tempfile::tempdir().unwrap();
let share = tmp.path().to_string_lossy().replace('\\', "/");
let hello = format!("{share}/hello.txt");
let cmd = vec![
"powershell".into(),
"-NoProfile".into(),
"-Command".into(),
format!("Set-Content -LiteralPath {hello} -Value hi"),
];
let mut config = MxcComputeConfig::default();
config.backend = MxcBackend::ProcessContainer;
config.egress_proxy = true;
config.egress_proxy_addr = "127.0.0.1:18080".into();
let backend = MxcComputeBackend::new_mocked(config);
let mut stream = backend.watch_sandboxes().await;
let mut policy = fs_policy(&[&share]);
policy.network_policies.insert(
"api".into(),
NetworkPolicyRule {
name: "api".into(),
endpoints: vec![NetworkEndpoint {
host: "example.com".into(),
ports: vec![443],
protocol: "rest".into(),
..Default::default()
}],
binaries: vec![NetworkBinary {
path: "/usr/bin/curl".into(),
..Default::default()
}],
},
);
let sandbox = with_policy(
driver_sandbox_with_command("sb-egress", &share, cmd),
policy.clone(),
);
backend
.create_sandbox(&sandbox)
.await
.expect("create accepted");
let ready = wait_for(&backend, "sb-egress", |s| {
ready_condition(s).is_some_and(|c| c.status == "True" && c.reason == "AgentRunning")
})
.await;
assert!(
ready.is_some(),
"egress split sandbox should reach Ready=True"
);
let recorded = crate::mxc::mock_recorded_config("sb-egress").expect("mock recorded config");
assert_eq!(recorded["network"]["defaultPolicy"], "block");
assert!(recorded["network"].get("allowedHosts").is_none());
assert!(recorded["network"].get("blockedHosts").is_none());
// MXC 0.6.0-alpha accepts only {"proxy": {"localhost": N}}.
let proxy_port = recorded["network"]["proxy"]["localhost"]
.as_u64()
.expect("proxy localhost port");
assert!(proxy_port > 0);
assert!(proxy_port <= u64::from(u16::MAX));
assert!(
recorded["network"]["proxy"].get("host").is_none(),
"proxy must not contain 'host' key"
);
assert!(
recorded["network"]["proxy"].get("port").is_none(),
"proxy must not contain 'port' key"
);
let reg = backend.registry.lock().await;
let entry = reg.get("sb-egress").expect("registry entry");
let entry_proxy_addr = entry.proxy_addr.expect("proxy addr");
assert_eq!(
entry_proxy_addr.ip(),
std::net::IpAddr::from([127, 0, 0, 1])
);
assert_eq!(u64::from(entry_proxy_addr.port()), proxy_port);
assert_eq!(
entry.trimmed_policy.as_ref().unwrap().network_policies,
policy.network_policies
);
drop(reg);
let mut saw_redirect = false;
let deadline = tokio::time::Instant::now() + Duration::from_secs(15);
while tokio::time::Instant::now() < deadline {
match tokio::time::timeout(Duration::from_millis(500), stream.next()).await {
Ok(Some(Ok(ev))) => {
if let Some(watch_sandboxes_event::Payload::PlatformEvent(pe)) = ev.payload
&& pe
.event
.as_ref()
.is_some_and(|e| e.reason == "EgressRedirect")
{
saw_redirect = true;
break;
}
}
Ok(_) => break,
Err(_) => continue,
}
}
assert!(saw_redirect, "expected EgressRedirect platform event");
}
#[tokio::test]
async fn negative_out_of_policy_write_is_denied_with_event() {
let share_tmp = tempfile::tempdir().unwrap();
@@ -1346,4 +1742,33 @@ mod lifecycle_tests {
assert_eq!(error.code(), tonic::Code::InvalidArgument);
assert!(backend.get_sandbox("sb-net").await.is_none());
}
#[tokio::test]
async fn governed_egress_rejects_network_middleware_before_lifecycle() {
let mut config = MxcComputeConfig::default();
config.egress_proxy = true;
config.egress_proxy_addr = "127.0.0.1:18080".into();
let backend = MxcComputeBackend::new_mocked(config);
let mut policy = fs_policy(&[]);
policy.network_middlewares.insert(
"redactor".into(),
NetworkMiddlewareConfig {
name: "redactor".into(),
middleware: "openshell/regex".into(),
on_error: "fail_closed".into(),
endpoints: Some(MiddlewareEndpointSelector {
include: vec!["api.example.com".into()],
exclude: Vec::new(),
}),
..Default::default()
},
);
let sandbox = with_policy(driver_sandbox("sb-middleware"), policy);
let error = backend.create_sandbox(&sandbox).await.unwrap_err();
assert_eq!(error.code(), tonic::Code::InvalidArgument);
assert!(error.message().contains("network_middlewares"));
assert!(backend.get_sandbox("sb-middleware").await.is_none());
}
}
crates/openshell-driver-mxc/src/mxc.rs
+10 -14
View File
@@ -99,10 +99,10 @@ fn network_json(network: &MxcNetwork) -> serde_json::Value {
// See also docs/reference/mxc-compute-driver-design.mdx §network.proxy.
// The MxcNetwork.proxy field remains SocketAddr so callers keep full
// precision; only the port is serialized into the localhost key.
// Released wxc-exec rejects allowedHosts and blockedHosts as unsupported,
// even when they are empty. The host proxy enforces the L7 allowlist.
let mut value = serde_json::json!({
"defaultPolicy": network.default_policy.as_str(),
"allowedHosts": [],
"blockedHosts": [],
});
if let Some(proxy) = network.proxy {
value["proxy"] = serde_json::json!({ "localhost": proxy.port() });
@@ -302,6 +302,10 @@ impl WxcExecInvoker {
}
}
pub(crate) const fn is_mock(&self) -> bool {
self.mock
}
/// Test-only constructor that forces mock mode without touching the
/// process-global `OPENSHELL_MXC_MOCK_WXC` env var (avoids races/UB across
/// parallel tests under edition 2024's `unsafe` `set_var`).
@@ -754,18 +758,8 @@ mod tests {
let config = provision_config_json(DEFAULT_CONFIGURATION_ID, &filesystem, Some(&network));
assert_eq!(config["network"]["defaultPolicy"], "block");
assert!(
config["network"]["allowedHosts"]
.as_array()
.unwrap()
.is_empty()
);
assert!(
config["network"]["blockedHosts"]
.as_array()
.unwrap()
.is_empty()
);
assert!(config["network"].get("allowedHosts").is_none());
assert!(config["network"].get("blockedHosts").is_none());
// MXC 0.6.0-alpha accepts only {"proxy": {"localhost": N}}.
assert_eq!(config["network"]["proxy"]["localhost"], 18080);
assert!(
@@ -788,6 +782,8 @@ mod tests {
proxy: Some("127.0.0.1:18080".parse().unwrap()),
};
let value = network_json(&network);
assert!(value.get("allowedHosts").is_none());
assert!(value.get("blockedHosts").is_none());
assert_eq!(value["proxy"]["localhost"], 18080);
assert!(value["proxy"].get("host").is_none());
assert!(value["proxy"].get("port").is_none());
crates/openshell-driver-mxc/src/policy.rs
+34 -1
View File
@@ -189,7 +189,9 @@ impl PolicyMapper for EmbeddedPolicyMapper {
#[cfg(test)]
mod tests {
use super::*;
use openshell_core::proto::FilesystemPolicy;
use openshell_core::proto::{
FilesystemPolicy, MiddlewareEndpointSelector, NetworkMiddlewareConfig,
};
fn demo_ctx() -> MapCtx {
MapCtx {
@@ -290,4 +292,35 @@ mod tests {
assert_eq!(trimmed.network_policies, policy.network_policies);
assert!(trimmed.filesystem.is_none());
}
#[test]
fn embedded_rejects_network_middleware_on_egress_proxy() {
let mapper = EmbeddedPolicyMapper;
let mut policy = fs_policy(&["C:/work/demo"], &[]);
policy.network_middlewares.insert(
"redactor".into(),
NetworkMiddlewareConfig {
name: "redactor".into(),
middleware: "openshell/regex".into(),
on_error: "fail_closed".into(),
endpoints: Some(MiddlewareEndpointSelector {
include: vec!["api.example.com".into()],
exclude: Vec::new(),
}),
..Default::default()
},
);
let ctx = MapCtx {
sandbox_id: "sb-egress-middleware".into(),
egress: Some("127.0.0.1:18080".parse().unwrap()),
};
let error = mapper.map(Some(&policy), &ctx).unwrap_err();
let MapError::Unsupported(loss) = error else {
panic!("expected unsupported middleware error");
};
assert_eq!(loss.len(), 1);
assert_eq!(loss[0].rule_kind, "network_middlewares");
assert!(loss[0].detail.contains("middleware service registry"));
}
}
crates/openshell-driver-mxc/src/policy_map/loss.rs
+1 -1
View File
@@ -1,7 +1,7 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
//! Loss-report model shared by the coarse map and the lossless split.
//! Loss-report model shared by the coarse map and the governed-egress split.
use std::collections::HashSet;
crates/openshell-driver-mxc/src/policy_map/map.rs
+28 -17
View File
@@ -21,7 +21,7 @@ use super::config::{
use super::loss::{LossItem, add_loss};
/// Options controlling the generated MXC config. Fields not relevant to the
/// coarse map (e.g. `proxy_redirect`) are reserved for the lossless split.
/// coarse map (e.g. `proxy_redirect`) are reserved for the governed-egress split.
#[derive(Clone, Debug)]
pub struct MxcMappingOptions {
/// MXC schema version written into `version`.
@@ -40,7 +40,7 @@ pub struct MxcMappingOptions {
pub timeout_ms: u64,
/// Emit `OpenShell` wildcard hosts into `allowedHosts` despite lossiness.
pub allow_wildcards: bool,
/// Governed-egress redirect address (used by the lossless split, not the
/// Governed-egress redirect address (used by the governed-egress split, not the
/// coarse map).
pub proxy_redirect: Option<SocketAddr>,
}
@@ -68,16 +68,16 @@ pub struct MxcMappingResult {
pub loss: Vec<LossItem>,
}
/// Result of the lossless split: the MXC config carries filesystem grants and a
/// Result of the governed-egress split: the MXC config carries filesystem grants and a
/// proxy redirect; the full network policy is returned unchanged for the
/// `OpenShell` CONNECT proxy to enforce.
#[derive(Clone, Debug)]
pub struct SplitPolicyResult {
/// MXC `ContainerConfig` with filesystem grants and `network.proxy` redirect.
///
/// `network.allowedHosts` is empty — direct egress is blocked at the MXC
/// layer. All outbound connections flow through the proxy; the proxy enforces
/// the full `OpenShell` network policy.
/// Direct egress is blocked at the MXC layer. Unsupported host-list fields
/// are omitted; all outbound connections flow through the proxy, which
/// enforces the full `OpenShell` network policy.
pub mxc_config: Value,
/// Full `OpenShell` network policy preserved verbatim for the host CONNECT
/// proxy. Only `network_policies` is populated; the proxy does not enforce
@@ -95,15 +95,16 @@ pub fn map_to_mxc(policy: &SandboxPolicy, opts: &MxcMappingOptions) -> MxcMappin
MxcMappingResult { config, loss }
}
/// Lossless split: map filesystem + containment to MXC, delegate network to the
/// Governed-egress split: map filesystem + containment to MXC, delegate network to the
/// `OpenShell` CONNECT proxy.
///
/// The returned [`SplitPolicyResult::mxc_config`] sets `network.proxy` to
/// `opts.proxy_redirect` and leaves `allowedHosts` empty — direct
/// egress is blocked at the MXC layer and all outbound connections flow through
/// the proxy. [`SplitPolicyResult::proxy_policy`] carries the original
/// `opts.proxy_redirect` and omits unsupported host-list fields. Direct egress
/// is blocked at the MXC layer and all outbound connections flow through the
/// proxy. [`SplitPolicyResult::proxy_policy`] carries the original
/// `network_policies` verbatim; no binary-scope, port, protocol, or wildcard
/// loss items are generated for the network side.
/// loss items are generated for those rules. Network middleware is rejected
/// until the host proxy can receive the gateway middleware service registry.
///
/// Returns `None` if `opts.proxy_redirect` is not set. Use [`map_to_mxc`]
/// for the standalone coarse path when no proxy is in the loop.
@@ -170,9 +171,23 @@ fn build_split_mxc_config(
"The host proxy receives the trimmed policy and enforces network rules.",
);
}
if !policy.network_middlewares.is_empty() {
add_loss(
items,
"network_middlewares",
"error",
&format!(
"{} network middleware config(s) cannot be enforced because the MXC host proxy is not connected to the gateway middleware service registry.",
policy.network_middlewares.len()
),
"network egress middleware",
"The MXC sandbox is rejected before launch instead of bypassing fail-open middleware or failing unrelated allowed traffic.",
);
}
// Direct egress is blocked; all outbound flows through the OpenShell proxy.
// allowedHosts is intentionally empty — the proxy enforces the full policy.
// Released wxc-exec rejects allowedHosts and blockedHosts as unsupported,
// even when empty, so the proxy path omits both fields.
//
// MXC 0.6.0-alpha schema accepts ONLY {"proxy": {"localhost": <port>}}.
// {"host": ..., "port": ...} and every other shape is rejected — verified
@@ -191,11 +206,7 @@ fn build_split_mxc_config(
"The redirect cannot be emitted; use a 127.0.0.1:PORT address.",
);
}
let mut network = json!({
"defaultPolicy": "block",
"allowedHosts": [],
"blockedHosts": [],
});
let mut network = json!({ "defaultPolicy": "block" });
if proxy_supported && proxy_addr.ip() == std::net::IpAddr::from([127, 0, 0, 1]) {
network["proxy"] = json!({ "localhost": proxy_addr.port() });
}
crates/openshell-driver-mxc/src/policy_map/mod.rs
+1 -1
View File
@@ -18,7 +18,7 @@
//! and anything MXC cannot express (ports, protocol, L7 rules, binary scope)
//! is recorded in the loss report. Use this when MXC enforces network on its
//! own, with no `OpenShell` proxy in the loop.
//! - [`split_policy`] — the *lossless* split for the Windows MXC compute
//! - [`split_policy`] — the governed-egress split for the Windows MXC compute
//! driver: MXC handles filesystem + containment + a `network.proxy` redirect,
//! while the full `OpenShell` network policy is preserved in a trimmed policy
//! enforced by the host CONNECT proxy.
crates/openshell-driver-mxc/tests/policy_mapper_examples.rs
+27 -12
View File
@@ -142,7 +142,7 @@ fn all_example_policies_map_with_invariants() {
}
#[test]
fn all_example_policies_split_with_lossless_invariants() {
fn all_example_policies_split_with_expected_invariants() {
let root = examples_root();
let mut policies = Vec::new();
discover(&root, &mut policies);
@@ -216,12 +216,28 @@ fn all_example_policies_split_with_lossless_invariants() {
assert_eq!(cfg["network"]["proxy"]["localhost"], 18080);
assert!(cfg["network"]["proxy"].get("host").is_none());
assert!(cfg["network"]["proxy"].get("port").is_none());
assert!(
result.loss.iter().all(|i| i.severity != "error"),
"processcontainer split must not emit error losses for {}: {:?}",
path.display(),
result.loss
);
let errors: Vec<_> = result
.loss
.iter()
.filter(|item| item.severity == "error")
.collect();
if policy.network_middlewares.is_empty() {
assert!(
errors.is_empty(),
"processcontainer split must not emit error losses for {}: {:?}",
path.display(),
result.loss
);
} else {
assert_eq!(
errors.len(),
1,
"middleware policy must have one fail-closed loss for {}: {:?}",
path.display(),
result.loss
);
assert_eq!(errors[0].path, "network_middlewares");
}
}
}
@@ -297,12 +313,11 @@ fn split_policy_routes_network_to_proxy() {
"non-127.0.0.1 redirect must produce an error loss item"
);
// Direct egress is blocked; allowedHosts is empty (proxy enforces the list).
// Direct egress is blocked; unsupported host-list fields are omitted and
// the proxy enforces the full list.
assert_eq!(cfg["network"]["defaultPolicy"], "block");
assert!(
str_list(&cfg["network"]["allowedHosts"]).is_empty(),
"split path must not populate allowedHosts"
);
assert!(cfg["network"].get("allowedHosts").is_none());
assert!(cfg["network"].get("blockedHosts").is_none());
// Filesystem grants are preserved unchanged.
assert_eq!(
crates/openshell-driver-mxc/tests/policy_mapper_matrix.rs
+6 -11
View File
@@ -393,11 +393,9 @@ fn a_split_proxy_localhost_port() {
result.mxc_config["network"]["proxy"]["localhost"], 18080,
"split must emit network.proxy.localhost == port"
);
// allowedHosts stays empty on the split path.
assert!(
str_list(&result.mxc_config["network"]["allowedHosts"]).is_empty(),
"split path must have empty allowedHosts"
);
// Released wxc-exec rejects host-list fields, even when empty.
assert!(result.mxc_config["network"].get("allowedHosts").is_none());
assert!(result.mxc_config["network"].get("blockedHosts").is_none());
}
// ─── QUADRANT B: OpenShell features MXC cannot express ──────────────────────
@@ -1009,7 +1007,7 @@ fn c_empty_policy_default_deny_posture() {
);
}
/// Split path with network rules present: allowedHosts stays empty.
/// Split path with network rules present: unsupported host lists stay absent.
#[test]
fn c_split_empty_allowed_hosts_with_network_rules() {
let mut policy = SandboxPolicy::default();
@@ -1025,11 +1023,8 @@ fn c_split_empty_allowed_hosts_with_network_rules() {
},
);
let result = split_policy(&policy, &pc_split_opts()).expect("split must return Some");
assert!(
str_list(&result.mxc_config["network"]["allowedHosts"]).is_empty(),
"split path allowedHosts must be empty even with network rules; got: {:?}",
result.mxc_config["network"]["allowedHosts"]
);
assert!(result.mxc_config["network"].get("allowedHosts").is_none());
assert!(result.mxc_config["network"].get("blockedHosts").is_none());
// But proxy redirect is present.
assert_eq!(
result.mxc_config["network"]["proxy"]["localhost"], 18080,
crates/openshell-driver-mxc/tests/wxc_exec_real.rs
+299 -94
View File
@@ -15,9 +15,9 @@
//! Two families:
//!
//! **(a) Dry-run contract tests** — exercise `--dry-run` only; pass/fail on
//! schema acceptance. Some `wxc-exec` builds select the DACL fallback during
//! dry-run and validate filesystem grants, so these tests use owned temporary
//! directories with concrete Windows paths.
//! schema acceptance. These pass on this box even though no enforcement
//! backend is live (dry-run validates the JSON schema without spinning up the
//! `AppContainer` or isolation session).
//!
//! **(b) Enforcement tests** — probe-gated; print a human-readable SKIP reason
//! and return early when the backend is not live. The probe distinguishes
@@ -31,8 +31,14 @@
#![cfg(target_os = "windows")]
use base64::Engine as _;
use openshell_core::proto::compute::v1::{DriverSandbox, DriverSandboxSpec, DriverSandboxTemplate};
use openshell_core::proto::{
FilesystemPolicy, NetworkBinary, NetworkEndpoint, NetworkPolicyRule, SandboxPolicy,
};
use openshell_driver_mxc::{MxcComputeBackend, MxcComputeConfig};
use std::path::PathBuf;
use std::process::Command;
use std::time::Duration;
// ── Path resolution ──────────────────────────────────────────────────────────
@@ -59,6 +65,17 @@ fn wxc_path() -> Option<PathBuf> {
None
}
/// Create a real, user-owned Windows directory for MXC filesystem grants.
///
/// MXC config values are literal paths: it does not expand `%TEMP%`. A unique
/// directory also keeps AppContainer+DACL fallback mutations scoped to test
/// data the current user owns.
fn temp_fixture() -> (tempfile::TempDir, String) {
let dir = tempfile::tempdir().expect("create MXC temp fixture");
let path = dir.path().to_string_lossy().into_owned();
(dir, path)
}
// ── Dry-run helper ────────────────────────────────────────────────────────────
/// Invoke `wxc-exec --config-base64 <cfg> --dry-run` synchronously.
@@ -94,19 +111,18 @@ fn dryrun_accepts_minimal_processcontainer_config() {
return;
};
let tmpdir = tempfile::tempdir().expect("tempdir");
let tmpdir_str = tmpdir.path().to_string_lossy().into_owned();
let (_tempdir, temp_path) = temp_fixture();
let config = serde_json::json!({
"version": "0.6.0-alpha",
"containerId": "test-minimal",
"containment": "processcontainer",
"process": {
"commandLine": "cmd /c exit 0",
"cwd": tmpdir_str,
"cwd": temp_path,
"timeout": 0,
},
"filesystem": {
"readwritePaths": [tmpdir_str],
"readwritePaths": [temp_path],
},
});
@@ -126,24 +142,21 @@ fn dryrun_accepts_network_block_without_proxy() {
return;
};
let tmpdir = tempfile::tempdir().expect("tempdir");
let tmpdir_str = tmpdir.path().to_string_lossy().into_owned();
let (_tempdir, temp_path) = temp_fixture();
let config = serde_json::json!({
"version": "0.6.0-alpha",
"containerId": "test-net-block",
"containment": "processcontainer",
"process": {
"commandLine": "cmd /c exit 0",
"cwd": tmpdir_str,
"cwd": temp_path,
"timeout": 0,
},
"filesystem": {
"readwritePaths": [tmpdir_str],
"readwritePaths": [temp_path],
},
"network": {
"defaultPolicy": "block",
"allowedHosts": [],
"blockedHosts": [],
},
});
@@ -165,24 +178,21 @@ fn dryrun_accepts_localhost_proxy_shape() {
return;
};
let tmpdir = tempfile::tempdir().expect("tempdir");
let tmpdir_str = tmpdir.path().to_string_lossy().into_owned();
let (_tempdir, temp_path) = temp_fixture();
let config = serde_json::json!({
"version": "0.6.0-alpha",
"containerId": "test-proxy-localhost",
"containment": "processcontainer",
"process": {
"commandLine": "cmd /c exit 0",
"cwd": tmpdir_str,
"cwd": temp_path,
"timeout": 0,
},
"filesystem": {
"readwritePaths": [tmpdir_str],
"readwritePaths": [temp_path],
},
"network": {
"defaultPolicy": "block",
"allowedHosts": [],
"blockedHosts": [],
"proxy": { "localhost": 18080 },
},
});
@@ -205,22 +215,21 @@ fn dryrun_rejects_host_port_proxy_shape() {
return;
};
let (_tempdir, temp_path) = temp_fixture();
let config = serde_json::json!({
"version": "0.6.0-alpha",
"containerId": "test-proxy-hostport",
"containment": "processcontainer",
"process": {
"commandLine": "cmd /c exit 0",
"cwd": "%TEMP%",
"cwd": temp_path,
"timeout": 0,
},
"filesystem": {
"readwritePaths": ["%TEMP%"],
"readwritePaths": [temp_path],
},
"network": {
"defaultPolicy": "block",
"allowedHosts": [],
"blockedHosts": [],
// MXC 0.6.0-alpha rejects {"host","port"} — verified empirically.
"proxy": { "host": "127.0.0.1", "port": 18080 },
},
@@ -243,17 +252,18 @@ fn dryrun_rejects_unknown_containment() {
return;
};
let (_tempdir, temp_path) = temp_fixture();
let config = serde_json::json!({
"version": "0.6.0-alpha",
"containerId": "test-bad-containment",
"containment": "nonsense",
"process": {
"commandLine": "cmd /c exit 0",
"cwd": "%TEMP%",
"cwd": temp_path,
"timeout": 0,
},
"filesystem": {
"readwritePaths": ["%TEMP%"],
"readwritePaths": [temp_path],
},
});
@@ -261,10 +271,9 @@ fn dryrun_rejects_unknown_containment() {
assert_ne!(code, 0, "unknown containment 'nonsense' should be rejected");
}
/// The most important dry-run test: parse the quickstart example policy with
/// `openshell_policy`, run `split_policy` (`proxy_redirect` 127.0.0.1:18080,
/// containment "processcontainer"), take the resulting `mxc_config`, inject a
/// real process block with a valid cwd, and verify that `--dry-run` exits 0.
/// The most important dry-run test: build a typed Windows policy, run
/// `split_policy` (proxy_redirect 127.0.0.1:18080, containment
/// "processcontainer"), and verify the resulting config with `--dry-run`.
///
/// This proves that the mapper's emitted JSON is accepted by the real binary —
/// the central contract of the policy-mapper integration.
@@ -276,33 +285,28 @@ fn dryrun_accepts_split_policy_output() {
return;
};
// Find the quickstart policy relative to CARGO_MANIFEST_DIR.
let manifest_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
let policy_path = manifest_dir.join("../../examples/sandbox-policy-quickstart/policy.yaml");
if !policy_path.exists() {
eprintln!(
"SKIP: quickstart policy not found at {}",
policy_path.display()
);
return;
}
let yaml = std::fs::read_to_string(&policy_path).expect("read policy YAML");
let policy = openshell_policy::parse_sandbox_policy(&yaml).expect("parse quickstart policy");
let (_tempdir, temp_path) = temp_fixture();
let policy = SandboxPolicy {
filesystem: Some(FilesystemPolicy {
include_workdir: false,
read_only: Vec::new(),
read_write: vec![temp_path.clone()],
}),
..Default::default()
};
let opts = openshell_driver_mxc::MxcMappingOptions {
containment: "processcontainer".to_string(),
command: "cmd /c exit 0".to_string(),
container_id: "split-policy-dryrun".to_string(),
cwd: Some(temp_path),
proxy_redirect: Some("127.0.0.1:18080".parse().unwrap()),
..Default::default()
};
let result = openshell_driver_mxc::split_policy(&policy, &opts)
.expect("split_policy must return Some when proxy_redirect is set");
// The quickstart policy has network_policies with error-level losses on
// isolation_session, but on processcontainer there should be zero error
// losses from the split itself. Warn if there are any error losses so the
// test is informative even when it proceeds.
// There should be no error losses on the processcontainer split path.
let error_losses: Vec<_> = result
.loss
.iter()
@@ -316,29 +320,7 @@ fn dryrun_accepts_split_policy_output() {
);
}
// Take the mapper's MXC config and inject the required process block.
// The split config does not include a process block (that comes from the
// gateway TOML at runtime); wxc-exec --dry-run requires one.
//
// The quickstart policy uses sandbox-internal Unix paths. Replace only the
// environment-dependent filesystem paths with an owned Windows directory:
// this test verifies the mapper's MXC JSON shape, while mapper unit tests
// cover the exact filesystem translation.
let tmpdir = tempfile::tempdir().expect("tempdir");
let tmpdir_str = tmpdir.path().to_string_lossy().into_owned();
let mut mxc_config = result.mxc_config.clone();
mxc_config["filesystem"] = serde_json::json!({
"readwritePaths": [tmpdir_str],
"readonlyPaths": [],
"deniedPaths": [],
});
mxc_config["process"] = serde_json::json!({
"commandLine": "cmd /c exit 0",
"cwd": tmpdir_str,
"timeout": 0,
});
// containerId is also required for processcontainer.
mxc_config["containerId"] = serde_json::json!("split-policy-dryrun");
let mxc_config = result.mxc_config;
let (code, stdout, stderr) = dry_run(&wxc, &mxc_config);
assert_eq!(
@@ -358,32 +340,39 @@ fn dryrun_accepts_split_policy_output() {
/// Probe the processcontainer backend.
///
/// Runs a trivial one-shot (`cmd /c exit 0`, owned temporary-directory grant).
/// Returns `Ok(())` when the backend is live, or `Err(reason)` when it is not (the
/// Runs a trivial one-shot (`cmd /c exit 0`, user-owned temp grant). Returns
/// `Ok(())` when the backend is live, or `Err(reason)` when it is not (the
/// caller prints SKIP + reason and returns from the test).
fn probe_processcontainer(wxc: &PathBuf) -> Result<(), String> {
// Abort early if the mock env var is set — a stale OPENSHELL_MXC_MOCK_WXC
// would silently turn this "real" run back into a mock run.
if std::env::var("OPENSHELL_MXC_MOCK_WXC").is_ok_and(|value| value == "1") {
if std::env::var("OPENSHELL_MXC_MOCK_WXC")
.map(|v| v == "1")
.unwrap_or(false)
{
return Err(
"OPENSHELL_MXC_MOCK_WXC=1 is set — unset it before running real enforcement tests"
.to_string(),
);
}
let tmpdir = tempfile::tempdir().map_err(|error| format!("tempdir failed: {error}"))?;
let tmpdir_str = tmpdir.path().to_string_lossy().into_owned();
let (_tempdir, temp_path) = temp_fixture();
let config = serde_json::json!({
"version": "0.6.0-alpha",
"containerId": "probe-pc",
"containment": "processcontainer",
"process": {
"commandLine": "cmd /c exit 0",
"cwd": tmpdir_str,
"timeout": 10,
"cwd": temp_path,
"timeout": 30_000,
},
"filesystem": {
"readwritePaths": [tmpdir_str],
"readwritePaths": [temp_path],
},
"ui": {
"disable": false,
"clipboard": "none",
"injection": false,
},
});
@@ -406,17 +395,16 @@ fn probe_processcontainer(wxc: &PathBuf) -> Result<(), String> {
|| combined.contains("not enabled")
{
// Extract the message if possible for a more useful skip reason.
let reason =
let reason = if let Ok(v) =
serde_json::from_str::<serde_json::Value>(&String::from_utf8_lossy(&out.stdout))
.map_or_else(
|_| "backend_error (velocity keys not enabled)".to_string(),
|value| {
value["error"]["message"]
.as_str()
.unwrap_or("backend_error (E_NOTIMPL)")
.to_string()
},
);
{
v["error"]["message"]
.as_str()
.unwrap_or("backend_error (E_NOTIMPL)")
.to_string()
} else {
"backend_error (velocity keys not enabled)".to_string()
};
return Err(reason);
}
@@ -432,12 +420,67 @@ fn probe_processcontainer(wxc: &PathBuf) -> Result<(), String> {
Ok(())
}
/// Probe the `isolation_session` backend.
/// Probe the released binary's live `network.proxy` support separately from
/// ordinary ProcessContainer support. Some builds accept the proxy JSON during
/// `--dry-run` but return `ERROR_INVALID_PARAMETER` from the live launcher.
fn probe_processcontainer_proxy(wxc: &PathBuf) -> Result<(), String> {
let (_tempdir, temp_path) = temp_fixture();
let proxy_listener = std::net::TcpListener::bind("127.0.0.1:0")
.map_err(|error| format!("failed to reserve proxy probe port: {error}"))?;
let proxy_port = proxy_listener
.local_addr()
.map_err(|error| format!("failed to read proxy probe port: {error}"))?
.port();
let config = serde_json::json!({
"version": "0.6.0-alpha",
"containerId": "probe-pc-proxy",
"containment": "processcontainer",
"process": {
"commandLine": "C:\\Windows\\System32\\cmd.exe /c exit 0",
"cwd": temp_path,
"timeout": 30_000,
},
"filesystem": {
"readwritePaths": [temp_path],
},
"processContainer": {
"leastPrivilege": false,
},
"network": {
"defaultPolicy": "block",
"proxy": { "localhost": proxy_port },
},
});
let json = serde_json::to_string(&config).unwrap();
let b64 = base64::engine::general_purpose::STANDARD.encode(json.as_bytes());
let output = Command::new(wxc)
.arg("--config-base64")
.arg(&b64)
.output()
.map_err(|error| format!("wxc-exec proxy probe failed to spawn: {error}"))?;
if output.status.success() {
return Ok(());
}
Err(format!(
"live network.proxy probe returned exit {}: stdout={} stderr={}",
output.status.code().unwrap_or(-1),
String::from_utf8_lossy(&output.stdout),
String::from_utf8_lossy(&output.stderr),
))
}
/// Probe the isolation_session backend.
///
/// Attempts a `provision` phase. Returns `Ok(sandbox_id)` when live, or
/// `Err(reason)` when the backend is unavailable (caller prints SKIP).
fn probe_isolation_session(wxc: &PathBuf) -> Result<String, String> {
if std::env::var("OPENSHELL_MXC_MOCK_WXC").is_ok_and(|value| value == "1") {
if std::env::var("OPENSHELL_MXC_MOCK_WXC")
.map(|v| v == "1")
.unwrap_or(false)
{
return Err(
"OPENSHELL_MXC_MOCK_WXC=1 is set — unset it before running real enforcement tests"
.to_string(),
@@ -586,7 +629,7 @@ fn pc_oneshot_in_policy_write_succeeds() {
"process": {
"commandLine": format!("cmd /c echo hello > \"{target_str}\""),
"cwd": tmpdir_str,
"timeout": 30,
"timeout": 30_000,
},
"filesystem": {
"readwritePaths": [tmpdir_str],
@@ -594,6 +637,11 @@ fn pc_oneshot_in_policy_write_succeeds() {
"processContainer": {
"leastPrivilege": false,
},
"ui": {
"disable": false,
"clipboard": "none",
"injection": false,
},
});
let json = serde_json::to_string(&config).unwrap();
@@ -619,8 +667,160 @@ fn pc_oneshot_in_policy_write_succeeds() {
);
}
/// Run an HTTPS request through the real driver and ProcessContainer. The
/// workload explicitly reads the injected bundle before curl uses it, proving
/// that the driver's internal TLS share is reachable from the AppContainer.
#[tokio::test]
#[ignore = "requires real wxc-exec and outbound HTTPS"]
async fn pc_https_egress_reads_injected_ca_bundle() {
let Some(wxc) = wxc_path() else {
eprintln!("SKIP: wxc-exec not found");
return;
};
if let Err(reason) = probe_processcontainer(&wxc) {
eprintln!("SKIP: processcontainer not live: {reason}");
return;
}
if let Err(reason) = probe_processcontainer_proxy(&wxc) {
eprintln!("SKIP: processcontainer network.proxy not live: {reason}");
return;
}
let system_root = std::env::var("SYSTEMROOT").expect("SYSTEMROOT must be set on Windows");
let cmd = PathBuf::from(&system_root).join("System32").join("cmd.exe");
let curl = PathBuf::from(system_root).join("System32").join("curl.exe");
if !curl.exists() {
eprintln!("SKIP: Windows curl.exe not found at {}", curl.display());
return;
}
let output_dir = tempfile::tempdir().expect("HTTPS output directory");
let output_path = output_dir.path().join("example.html");
let certificate_path = output_dir.path().join("peer-certificate.txt");
let output_dir_string = output_dir.path().to_string_lossy().into_owned();
let output_path_string = output_path.to_string_lossy().into_owned();
let certificate_path_string = certificate_path.to_string_lossy().into_owned();
let cmd_string = cmd.to_string_lossy().into_owned();
let script = format!(
"type \"%CURL_CA_BUNDLE%\" 1>NUL && \
\"{}\" --fail --silent --show-error --cacert \"%CURL_CA_BUNDLE%\" \
https://example.com/ --output \"{output_path_string}\" \
--write-out \"%{{certs}}\" 1>\"{certificate_path_string}\"",
curl.display()
);
let command = vec![
cmd_string.clone(),
"/d".to_string(),
"/c".to_string(),
script,
];
let serde_json::Value::Object(driver_config) = serde_json::json!({
"command": command,
"cwd": output_dir_string,
}) else {
unreachable!();
};
let policy = SandboxPolicy {
version: 1,
filesystem: Some(FilesystemPolicy {
include_workdir: false,
read_only: Vec::new(),
read_write: vec![output_dir_string],
}),
network_policies: std::collections::HashMap::from([(
"https_example".to_string(),
NetworkPolicyRule {
name: "https-example".to_string(),
endpoints: vec![NetworkEndpoint {
host: "example.com".to_string(),
ports: vec![443],
protocol: "rest".to_string(),
tls: "terminate".to_string(),
enforcement: "enforce".to_string(),
access: "read-only".to_string(),
..Default::default()
}],
binaries: vec![NetworkBinary {
path: cmd_string,
..Default::default()
}],
},
)]),
..Default::default()
};
let sandbox = DriverSandbox {
id: "pc-https-ca".to_string(),
name: "pc-https-ca".to_string(),
spec: Some(DriverSandboxSpec {
template: Some(DriverSandboxTemplate {
driver_config: Some(
openshell_core::proto_struct::json_object_to_struct(driver_config)
.expect("driver config"),
),
..Default::default()
}),
policy: Some(policy),
..Default::default()
}),
..Default::default()
};
let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();
let config = MxcComputeConfig {
wxc_exec_path: wxc.to_string_lossy().into_owned(),
egress_proxy: true,
egress_proxy_addr: "127.0.0.1:18080".to_string(),
..Default::default()
};
let backend = MxcComputeBackend::new(config);
backend
.create_sandbox(&sandbox)
.await
.expect("real HTTPS sandbox create accepted");
let mut terminal_condition = None;
for _ in 0..600 {
if let Some(observed) = backend.get_sandbox("pc-https-ca").await
&& let Some(condition) = observed
.status
.and_then(|status| status.conditions.into_iter().find(|c| c.r#type == "Ready"))
&& matches!(
condition.reason.as_str(),
"AgentCompleted" | "ExecFailed" | "ProvisionFailed"
)
{
terminal_condition = Some(condition);
break;
}
tokio::time::sleep(Duration::from_millis(100)).await;
}
let condition = terminal_condition.expect("HTTPS sandbox should reach a terminal condition");
assert_eq!(
condition.reason, "AgentCompleted",
"HTTPS workload failed: {}",
condition.message
);
assert!(output_path.exists(), "curl should write the HTTPS response");
assert!(
std::fs::metadata(&output_path)
.expect("HTTPS response metadata")
.len()
> 0,
"HTTPS response should not be empty"
);
let peer_certificate =
std::fs::read_to_string(certificate_path).expect("curl peer certificate output");
assert!(
peer_certificate.contains("OpenShell Sandbox CA"),
"HTTPS response must use a certificate issued by the host proxy CA"
);
}
/// Write to a path OUTSIDE the granted dir; assert exit non-zero and file absent.
/// This is the genuine OS default-deny proof — the `AppContainer` blocks the write
/// This is the genuine OS default-deny proof — the AppContainer blocks the write
/// without requiring any host ACL lockdown. The mock can only fake this.
#[test]
#[ignore = "requires real wxc-exec"]
@@ -648,7 +848,7 @@ fn pc_oneshot_out_of_policy_write_denied() {
"process": {
"commandLine": format!("cmd /c echo denied > \"{denied_file_str}\""),
"cwd": granted_str,
"timeout": 30,
"timeout": 30_000,
},
"filesystem": {
// Only the granted_dir is in policy — denied_dir is NOT granted.
@@ -657,6 +857,11 @@ fn pc_oneshot_out_of_policy_write_denied() {
"processContainer": {
"leastPrivilege": false,
},
"ui": {
"disable": false,
"clipboard": "none",
"injection": false,
},
});
let json = serde_json::to_string(&config).unwrap();
@@ -685,7 +890,7 @@ fn pc_oneshot_out_of_policy_write_denied() {
// ── Isolation session enforcement tests ──────────────────────────────────────
/// Full `isolation_session` round trip: provision → start → exec → stop →
/// Full isolation_session round trip: provision → start → exec → stop →
/// deprovision. `deprovision` runs in a drop-guard even on panic so the
/// single-session backend is never left orphaned.
#[test]
crates/openshell-ocsf/tests/event_identity.rs
+6 -1
View File
@@ -68,7 +68,12 @@ fn device_identifies_the_sandbox_environment() {
assert_eq!(json["device"]["type_id"], 99);
assert_eq!(json["device"]["type"], "Sandbox");
assert_eq!(json["device"]["os"]["name"], "Linux");
let expected_os = if cfg!(target_os = "windows") {
"Windows"
} else {
"Linux"
};
assert_eq!(json["device"]["os"]["name"], expected_os);
}
#[test]
crates/openshell-sandbox/src/endpoint_status.rs
+1194
View File
File diff suppressed because it is too large Load Diff
crates/openshell-sandbox/src/lib.rs
+131 -3
View File
@@ -17,6 +17,7 @@ compile_error!(
mod activity_aggregator;
mod denial_aggregator;
mod endpoint_status;
#[cfg_attr(not(target_os = "linux"), allow(dead_code))]
mod google_cloud_metadata;
mod mechanistic_mapper;
@@ -69,6 +70,10 @@ use openshell_ocsf::{
pub(crate) use openshell_ocsf::ctx::ctx as ocsf_ctx;
use openshell_core::denial::DenialEvent;
use openshell_core::endpoint_status::{
EndpointConfigVersion, EndpointObservationSender, EndpointStatusReceiver,
EndpointStatusSnapshot, endpoint_status_channel,
};
use openshell_core::policy::{NetworkMode, NetworkPolicy, ProxyPolicy, SandboxPolicy};
use openshell_core::proposals::AgentProposals;
use openshell_core::provider_credentials::ProviderCredentialState;
@@ -78,7 +83,7 @@ use openshell_supervisor_process::process::ProcessEnforcementMode;
pub use openshell_supervisor_process::process::{ProcessHandle, ProcessStatus};
use openshell_supervisor_process::skills;
use tokio::sync::mpsc::UnboundedSender;
#[cfg(any(test, target_os = "linux"))]
#[cfg(any(target_os = "linux", test))]
use tokio::time::timeout;
const SIDECAR_NETWORK_ENFORCEMENT_MODE: &str = "sidecar-nftables";
@@ -751,6 +756,46 @@ pub async fn run_sandbox(
#[cfg(not(target_os = "linux"))]
drop(bypass_activity_tx);
// Endpoint observations use a bounded channel and never add backpressure
// to proxied traffic. Queue the reset only after the corresponding policy
// and provider state have both been installed during startup.
let (endpoint_observation_tx, endpoint_status_rx) = if sandbox_id.is_some() && network_enabled {
let (sender, receiver) = endpoint_status_channel();
(Some(sender), Some(receiver))
} else {
(None, None)
};
// Endpoint observations are authorized by the currently accepted supervisor
// stream, not merely by the sandbox's long-lived bearer credential.
let (supervisor_session_updates, supervisor_session_id) =
tokio::sync::watch::channel::<Option<String>>(None);
if let (
Some(sender),
Some(proto),
LoadedPolicyOrigin::Gateway {
revision: Some(revision),
..
},
) = (
endpoint_observation_tx.as_ref(),
retained_proto.as_ref(),
&loaded_policy_origin,
) {
let config_version = EndpointConfigVersion {
policy_hash: revision.policy_hash.clone(),
provider_env_revision: provider_credentials.snapshot().revision,
};
if let Err(error) = sender
.reset(
config_version,
endpoint_status::mcp_endpoint_inventory(proto),
)
.await
{
warn!(%error, "Endpoint status tracker unavailable during startup");
}
}
// Workspace watch: the policy poll loop learns the workspace from
// GetSandboxConfig and broadcasts it. Flush tasks and the policy.local
// API read the current value so proposals target the correct workspace.
@@ -780,6 +825,7 @@ pub async fn run_sandbox(
openshell_endpoint_for_proxy.as_deref(),
denial_tx,
activity_tx,
endpoint_observation_tx.clone(),
agent_proposals.clone(),
workspace_rx.clone(),
&upstream_proxy_args,
@@ -864,6 +910,7 @@ pub async fn run_sandbox(
config_apply_tx: config_apply_tx.clone(),
prepared_supervisor_session: prepared_supervisor_session.take(),
prepared_bootstrap_result: prepared_bootstrap_result.take(),
supervisor_session_updates: supervisor_session_updates.clone(),
},
);
}
@@ -982,6 +1029,10 @@ pub async fn run_sandbox(
let poll_pid = entrypoint_pid.clone();
let poll_provider_credentials = provider_credentials.clone();
let poll_policy_local = networking.as_ref().map(|n| n.policy_local_ctx.clone());
let poll_endpoint_policy = loaded_policy_origin
.allows_gateway_policy_reload()
.then(|| retained_proto.clone())
.flatten();
let poll_interval_secs: u64 = std::env::var("OPENSHELL_POLICY_POLL_INTERVAL_SECS")
.ok()
.and_then(|v| v.parse().ok())
@@ -1013,6 +1064,10 @@ pub async fn run_sandbox(
.as_ref()
.and_then(|bootstrap| bootstrap.sandbox_config.clone())
.map(Into::into),
endpoint_observation_tx,
endpoint_status_rx,
endpoint_policy: poll_endpoint_policy,
supervisor_session_id,
};
tokio::spawn(async move {
@@ -1208,6 +1263,7 @@ pub async fn run_sandbox(
entrypoint_pid,
entrypoint_started_tx,
sidecar_exit_tx,
Some(supervisor_session_updates),
provider_credentials,
main_env,
ca_file_paths,
@@ -1577,6 +1633,7 @@ struct SidecarEntrypointHandler {
prepared_supervisor_session:
Option<openshell_supervisor_process::supervisor_session::PreparedSupervisorSession>,
prepared_bootstrap_result: Option<openshell_core::proto::ConfigBootstrapResult>,
supervisor_session_updates: tokio::sync::watch::Sender<Option<String>>,
}
#[cfg(target_os = "linux")]
@@ -1596,6 +1653,7 @@ fn spawn_sidecar_entrypoint_handler(
config_apply_tx,
mut prepared_supervisor_session,
mut prepared_bootstrap_result,
supervisor_session_updates,
} = handler;
let mut session_started = false;
let mut session_task: Option<tokio::task::JoinHandle<()>> = None;
@@ -1709,6 +1767,7 @@ fn spawn_sidecar_entrypoint_handler(
Some(supervisor_pid),
Arc::clone(&terminating),
config_apply_tx.clone(),
Some(supervisor_session_updates.clone()),
),
)
} else {
@@ -1719,8 +1778,11 @@ fn spawn_sidecar_entrypoint_handler(
None,
Some(supervisor_pid),
Arc::clone(&terminating),
started.instance_id.clone(),
Some(config_apply_tx.clone()),
openshell_supervisor_process::supervisor_session::SessionRuntimeContext {
instance_id: started.instance_id.clone(),
session_id_updates: Some(supervisor_session_updates.clone()),
config_apply_tx: Some(config_apply_tx.clone()),
},
))
};
session_started = true;
@@ -3480,6 +3542,14 @@ trait PolicyGatewayClient: Clone + Send + Sync + 'static {
error: &str,
) -> Result<()>;
async fn report_endpoint_status(
&self,
_sandbox_id: &str,
_snapshot: &EndpointStatusSnapshot,
) -> Result<()> {
Ok(())
}
async fn refresh_installed_extension_credentials(&self) -> Result<()> {
Ok(())
}
@@ -3514,6 +3584,14 @@ impl PolicyGatewayClient for openshell_core::grpc_client::CachedOpenShellClient
.await
}
async fn report_endpoint_status(
&self,
sandbox_id: &str,
snapshot: &EndpointStatusSnapshot,
) -> Result<()> {
self.report_endpoint_status(sandbox_id, snapshot).await
}
async fn refresh_installed_extension_credentials(&self) -> Result<()> {
self.refresh_installed_extension_credentials().await
}
@@ -3702,6 +3780,14 @@ struct PolicyPollLoopContext {
/// runtime state, so this seeds equality tracking and disables fetch-based
/// reconciliation for the current protocol.
initial_stream_snapshot: Option<openshell_core::grpc_client::SettingsPollResult>,
/// Producer shared with network enforcement and policy installation.
endpoint_observation_tx: Option<EndpointObservationSender>,
/// Single FIFO consumed by the endpoint status reporter.
endpoint_status_rx: Option<EndpointStatusReceiver>,
/// Gateway policy currently installed in OPA, absent for local overrides.
endpoint_policy: Option<openshell_core::proto::SandboxPolicy>,
/// Session accepted by `ConnectSupervisor`; `None` suspends endpoint reports.
supervisor_session_id: tokio::sync::watch::Receiver<Option<String>>,
}
type MiddlewareConnector = Arc<
@@ -4545,6 +4631,14 @@ async fn run_policy_poll_loop_with_client<C: PolicyGatewayClient>(
ctx.sandbox_id.clone(),
status_receiver,
));
if let Some(endpoint_status_receiver) = ctx.endpoint_status_rx.take() {
tokio::spawn(endpoint_status::run_reporter(
client.clone(),
ctx.sandbox_id.clone(),
endpoint_status_receiver,
ctx.supervisor_session_id.clone(),
));
}
let initial_stream_snapshot = ctx.initial_stream_snapshot.take();
let stream_authoritative = initial_stream_snapshot.is_some();
@@ -4585,6 +4679,7 @@ async fn run_policy_poll_loop_with_client<C: PolicyGatewayClient>(
let mut last_failed_runtime_revision: Option<FailedRuntimeRevision> = None;
let mut rejected_policy_generation: Option<RejectedPolicyGeneration> = None;
let mut has_last_valid_policy = ctx.loaded_policy_origin.has_last_valid_policy();
let mut current_endpoint_policy = ctx.endpoint_policy.clone();
// A first poll that does not match the policy already loaded into OPA must
// pass through the normal reconciliation path immediately. It must never
@@ -4598,6 +4693,13 @@ async fn run_policy_poll_loop_with_client<C: PolicyGatewayClient>(
let _ = ctx.workspace_tx.send(snapshot.workspace.clone());
apply_ocsf_json_setting(&ctx.ocsf_enabled, &snapshot.settings);
apply_ocsf_schema_version_setting(&ctx.ocsf_schema_version, &snapshot.settings);
endpoint_status::reset(
ctx.endpoint_observation_tx.as_ref(),
current_endpoint_policy.as_ref(),
&current_policy_hash,
ctx.provider_credentials.snapshot().revision,
)
.await;
} else {
match client.poll_settings(&ctx.sandbox_id).await {
Ok(result) => {
@@ -4621,6 +4723,14 @@ async fn run_policy_poll_loop_with_client<C: PolicyGatewayClient>(
current_config_revision = candidate.config_revision;
current_policy_version = candidate.version;
current_policy_hash.clone_from(&candidate.policy_hash);
current_endpoint_policy.clone_from(&result.policy);
endpoint_status::reset(
ctx.endpoint_observation_tx.as_ref(),
current_endpoint_policy.as_ref(),
&current_policy_hash,
ctx.provider_credentials.snapshot().revision,
)
.await;
current_middleware_services = result.supervisor_middleware_services;
current_extension_authentication_enabled =
result.extension_authentication_enabled;
@@ -4813,6 +4923,7 @@ async fn run_policy_poll_loop_with_client<C: PolicyGatewayClient>(
&result,
);
let mut policy_runtime_reconciled = false;
let mut provider_runtime_changed = false;
// A local policy override is not coupled to the gateway policy
// snapshot, so its service registry can still be reconciled alone.
@@ -4908,6 +5019,7 @@ async fn run_policy_poll_loop_with_client<C: PolicyGatewayClient>(
env_result.static_credential_bindings,
env_result.non_secret_environment_keys,
);
provider_runtime_changed = true;
if let Err(error) = install_result {
ocsf_emit!(
ConfigStateChangeBuilder::new(ocsf_ctx())
@@ -4946,6 +5058,7 @@ async fn run_policy_poll_loop_with_client<C: PolicyGatewayClient>(
Err(e) => {
ctx.provider_credentials
.revoke_static_provider_environment(result.provider_env_revision);
provider_runtime_changed = true;
warn!(
error = %e,
provider_env_revision = result.provider_env_revision,
@@ -5078,6 +5191,7 @@ async fn run_policy_poll_loop_with_client<C: PolicyGatewayClient>(
}
current_policy_hash.clone_from(&result.policy_hash);
current_endpoint_policy.clone_from(&result.policy);
current_middleware_services.clone_from(&result.supervisor_middleware_services);
current_extension_authentication_enabled =
result.extension_authentication_enabled;
@@ -5187,6 +5301,16 @@ async fn run_policy_poll_loop_with_client<C: PolicyGatewayClient>(
current_policy_version = version;
}
if policy_runtime_reconciled || provider_runtime_changed {
endpoint_status::reset(
ctx.endpoint_observation_tx.as_ref(),
current_endpoint_policy.as_ref(),
&current_policy_hash,
ctx.provider_credentials.snapshot().revision,
)
.await;
}
// Apply OCSF JSON toggle from the `ocsf_json_enabled` setting.
apply_ocsf_json_setting(&ctx.ocsf_enabled, &result.settings);
apply_ocsf_schema_version_setting(&ctx.ocsf_schema_version, &result.settings);
@@ -5992,6 +6116,10 @@ network_policies:
transparent_tcp: TransparentTcpReloadState::default(),
config_apply_rx: None,
initial_stream_snapshot: None,
endpoint_observation_tx: None,
endpoint_status_rx: None,
endpoint_policy: None,
supervisor_session_id: tokio::sync::watch::channel(None).1,
}
}
crates/openshell-sdk/tests/client_mock.rs
+7
View File
@@ -157,6 +157,13 @@ fn workload_template_proto(name: &str, workspace: &str) -> proto::SandboxWorkloa
#[tonic::async_trait]
impl OpenShell for TestOpenShell {
async fn report_endpoint_status(
&self,
_request: tonic::Request<proto::ReportEndpointStatusRequest>,
) -> Result<Response<proto::ReportEndpointStatusResponse>, Status> {
Ok(Response::new(proto::ReportEndpointStatusResponse {}))
}
async fn begin_rootfs_tar_staging(
&self,
_request: tonic::Request<proto::BeginRootfsTarStagingRequest>,
crates/openshell-server/Cargo.toml
+1
View File
@@ -65,6 +65,7 @@ clap = { workspace = true }
miette = { workspace = true }
thiserror = { workspace = true }
anyhow = { workspace = true }
chrono = { version = "0.4", default-features = false, features = ["clock", "std"] }
# Logging
tracing = { workspace = true }
crates/openshell-server/src/certgen.rs
+194 -2
View File
@@ -24,7 +24,7 @@
use clap::Args;
use k8s_openapi::ByteString;
use k8s_openapi::api::core::v1::Secret;
use k8s_openapi::api::core::v1::{ConfigMap, Secret};
use kube::Client;
use kube::api::{Api, ObjectMeta, PostParams};
use miette::{IntoDiagnostic, Result, WrapErr};
@@ -78,6 +78,33 @@ pub struct CertgenArgs {
/// For local debugging.
#[arg(long)]
dry_run: bool,
/// Name of a `ConfigMap` to create containing the CA certificate (key: ca.crt)
/// for `BackendTLSPolicy` backend validation. The CA is always read from the
/// authoritative server Secret: --server-secret-name in full PKI mode,
/// --backend-ca-source-secret in --jwt-only mode.
#[arg(long, value_name = "NAME")]
backend_ca_configmap_name: Option<String>,
/// Name of an existing `Secret` containing a ca.crt key to populate the
/// backend CA `ConfigMap` from. Required with --jwt-only when
/// --backend-ca-configmap-name is set (typically the server TLS `Secret`
/// created by cert-manager).
#[arg(long, value_name = "NAME", requires = "backend_ca_configmap_name")]
backend_ca_source_secret: Option<String>,
/// Maximum time in seconds to poll for the backend CA source `Secret` when
/// using cert-manager. Defaults to 90 seconds. The Helm chart sets this to
/// (Job activeDeadlineSeconds - 30) to leave margin for `ConfigMap` creation.
#[arg(long, value_name = "SECONDS", default_value = "90")]
backend_ca_poll_timeout_seconds: u64,
/// Fail with an error if the backend CA source `Secret` is not found within
/// the polling timeout. When false (default), the hook succeeds with a warning
/// and the `ConfigMap` is not created. The Helm chart sets this based on
/// pkiInitJob.failOnTimeout.
#[arg(long)]
backend_ca_fail_on_timeout: bool,
}
pub async fn run(args: CertgenArgs) -> Result<()> {
@@ -97,7 +124,13 @@ pub async fn run(args: CertgenArgs) -> Result<()> {
run_local(dir, &args.server_sans)
} else {
let bundle = generate_pki(&args.server_sans)?;
run_kubernetes(&args, &bundle).await
run_kubernetes(&args, &bundle).await?;
if let Some(ref cm_name) = args.backend_ca_configmap_name {
create_backend_ca_configmap_if_needed(&args, cm_name).await?;
}
Ok(())
}
}
@@ -293,6 +326,165 @@ async fn create_tls_secrets(
Ok(())
}
fn extract_ca_from_secret(secret: &Secret, name: &str) -> Result<String> {
let data = secret
.data
.as_ref()
.ok_or_else(|| miette::miette!("secret {name} has no data"))?;
let ca = data
.get("ca.crt")
.ok_or_else(|| miette::miette!("secret {name} has no ca.crt key"))?;
String::from_utf8(ca.0.clone())
.into_diagnostic()
.wrap_err("ca.crt is not valid UTF-8")
}
async fn create_backend_ca_configmap_if_needed(
args: &CertgenArgs,
configmap_name: &str,
) -> Result<()> {
let namespace = args
.namespace
.as_deref()
.ok_or_else(|| miette::miette!("--namespace is required (or set POD_NAMESPACE)"))?;
let client = Client::try_default()
.await
.into_diagnostic()
.wrap_err("failed to construct Kubernetes client for backend CA ConfigMap")?;
let secret_api: Api<Secret> = Api::namespaced(client.clone(), namespace);
let cm_api: Api<ConfigMap> = Api::namespaced(client, namespace);
// Resolve the CA from the authoritative server Secret rather than the
// in-memory bundle so upgrades that enable BackendTLSPolicy on an
// existing release use the CA that actually signed the server cert.
let ca_pem = if let Some(source_secret) = &args.backend_ca_source_secret {
let poll_timeout = std::time::Duration::from_secs(args.backend_ca_poll_timeout_seconds);
let poll_interval = std::time::Duration::from_secs(2);
let start = std::time::Instant::now();
let secret = loop {
match secret_api
.get_opt(source_secret)
.await
.into_diagnostic()
.wrap_err_with(|| format!("failed to read secret {source_secret}"))?
{
Some(secret) => break secret,
None if start.elapsed() >= poll_timeout => {
let msg = format!(
"Backend CA source secret {source_secret} not found after {timeout_secs}s; \
ConfigMap {configmap_name} not created. This is expected if cert-manager \
is still issuing the certificate.",
timeout_secs = poll_timeout.as_secs()
);
if args.backend_ca_fail_on_timeout {
return Err(miette::miette!(
"{msg} Install failed due to --backend-ca-fail-on-timeout."
));
}
warn!(
secret = %source_secret,
configmap = %configmap_name,
timeout_secs = poll_timeout.as_secs(),
"{msg} Run helm upgrade after the TLS secret exists or the BackendTLSPolicy \
will remain non-functional until the ConfigMap is created manually."
);
return Ok(());
}
None => {
if start.elapsed().as_secs().is_multiple_of(10) {
info!(
secret = %source_secret,
elapsed_secs = start.elapsed().as_secs(),
"Waiting for cert-manager to issue TLS certificate..."
);
}
tokio::time::sleep(poll_interval).await;
}
}
};
info!(
secret = %source_secret,
elapsed_secs = start.elapsed().as_secs(),
"cert-manager TLS certificate found."
);
extract_ca_from_secret(&secret, source_secret)?
} else if let Some(server_secret) = &args.server_secret_name {
let secret = secret_api
.get(server_secret)
.await
.into_diagnostic()
.wrap_err_with(|| format!("failed to read server secret {server_secret}"))?;
extract_ca_from_secret(&secret, server_secret)?
} else {
return Err(miette::miette!(
"--backend-ca-source-secret or --server-secret-name is required \
with --backend-ca-configmap-name"
));
};
// Reconcile: create or update so the backend CA stays current across
// CA rotations and upgrades.
if let Some(existing) = cm_api
.get_opt(configmap_name)
.await
.into_diagnostic()
.wrap_err_with(|| format!("failed to read configmap {configmap_name}"))?
{
let up_to_date = existing
.data
.as_ref()
.and_then(|d| d.get("ca.crt"))
.map(String::as_str)
== Some(&ca_pem);
if up_to_date {
info!(
namespace = %namespace,
configmap = %configmap_name,
"Backend CA ConfigMap is up-to-date, skipping."
);
return Ok(());
}
let mut updated = existing;
updated.data = Some(BTreeMap::from([("ca.crt".to_string(), ca_pem)]));
cm_api
.replace(configmap_name, &PostParams::default(), &updated)
.await
.into_diagnostic()
.wrap_err_with(|| format!("failed to update configmap {configmap_name}"))?;
info!(
namespace = %namespace,
configmap = %configmap_name,
"Backend CA ConfigMap updated with current CA."
);
return Ok(());
}
let configmap = ConfigMap {
metadata: ObjectMeta {
name: Some(configmap_name.to_string()),
..Default::default()
},
data: Some(BTreeMap::from([("ca.crt".to_string(), ca_pem)])),
..Default::default()
};
cm_api
.create(&PostParams::default(), &configmap)
.await
.into_diagnostic()
.wrap_err_with(|| format!("failed to create configmap {configmap_name}"))?;
info!(
namespace = %namespace,
configmap = %configmap_name,
"Backend CA ConfigMap created."
);
Ok(())
}
fn tls_secret(name: &str, crt_pem: &str, key_pem: &str, ca_pem: &str) -> Secret {
let mut data = BTreeMap::new();
data.insert(
crates/openshell-server/src/cli.rs
+53
View File
@@ -2314,6 +2314,59 @@ mod tests {
.expect("complete package-generated TLS paths may not exist before certificate generation");
}
#[test]
fn generate_certs_backend_ca_configmap_flags_parse() {
let _lock = ENV_LOCK
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let _g1 = EnvVarGuard::remove("OPENSHELL_DB_URL");
let _g2 = EnvVarGuard::remove("POD_NAMESPACE");
let cli = Cli::try_parse_from([
"openshell-gateway",
"generate-certs",
"--namespace",
"openshell",
"--jwt-only",
"--jwt-secret-name",
"openshell-jwt-keys",
"--backend-ca-configmap-name",
"openshell-backend-ca",
"--backend-ca-source-secret",
"openshell-server-tls",
])
.expect("backend CA ConfigMap flags should parse with --jwt-only");
assert!(matches!(
cli.command,
Some(super::Commands::GenerateCerts(_))
));
}
#[test]
fn generate_certs_backend_ca_source_secret_requires_configmap_name() {
let _lock = ENV_LOCK
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let _g1 = EnvVarGuard::remove("OPENSHELL_DB_URL");
let _g2 = EnvVarGuard::remove("POD_NAMESPACE");
let err = Cli::try_parse_from([
"openshell-gateway",
"generate-certs",
"--namespace",
"openshell",
"--jwt-only",
"--jwt-secret-name",
"openshell-jwt-keys",
"--backend-ca-source-secret",
"openshell-server-tls",
])
.expect_err("--backend-ca-source-secret should require --backend-ca-configmap-name");
assert_eq!(err.kind(), clap::error::ErrorKind::MissingRequiredArgument);
}
#[test]
fn bare_invocation_with_no_db_url_parses_for_runtime_defaults() {
// db_url is Option<String> at the clap level so subcommand parsing
crates/openshell-server/src/compute/mod.rs
+41
View File
@@ -4282,6 +4282,7 @@ fn public_status_from_driver(
current_policy_version,
main_process_instance_id: String::new(),
exit_code: None,
endpoint_statuses: Vec::new(),
}
}
@@ -4291,6 +4292,13 @@ fn apply_driver_snapshot(
session_connected: bool,
driver_reports_runtime_readiness: bool,
) {
// Endpoint results belong to the gateway. A driver reports infrastructure
// and runtime state, so a full driver snapshot must preserve these records.
let endpoint_statuses = sandbox
.status
.as_ref()
.map(|status| status.endpoint_statuses.clone())
.unwrap_or_default();
let old_phase = SandboxPhase::try_from(sandbox.phase()).unwrap_or(SandboxPhase::Unknown);
let sandbox_name = &incoming.name;
@@ -4367,6 +4375,7 @@ fn apply_driver_snapshot(
if let Some(status) = status.as_mut() {
status.phase = phase as i32;
status.endpoint_statuses = endpoint_statuses;
}
if let Some(status) = status.as_mut()
@@ -6451,6 +6460,38 @@ mod tests {
}
}
#[test]
fn driver_snapshot_preserves_endpoint_failure_and_ready_phase() {
let mut sandbox = sandbox_record("sandbox-id", "sandbox-name", SandboxPhase::Ready);
let endpoint = openshell_core::proto::EndpointStatus {
endpoint_id: "endpoint:v1:gateway".to_string(),
host: "api.example.com".to_string(),
ports: vec![443],
path: "/mcp".to_string(),
last_result: openshell_core::proto::EndpointResult::TransportFailed as i32,
last_reported_at: "2026-09-05T01:01:00.000Z".to_string(),
};
sandbox.status = Some(SandboxStatus {
sandbox_name: "sandbox-name".to_string(),
phase: SandboxPhase::Ready as i32,
endpoint_statuses: vec![endpoint.clone()],
..Default::default()
});
let incoming = ready_driver_sandbox("sandbox-id", "sandbox-name");
apply_driver_snapshot(&mut sandbox, &incoming, true, true);
assert_eq!(sandbox.phase(), SandboxPhase::Ready as i32);
let status = sandbox.status.expect("driver status applied");
assert_eq!(status.endpoint_statuses, vec![endpoint]);
assert!(
status
.conditions
.iter()
.any(|condition| { condition.r#type == "Ready" && condition.status == "True" })
);
}
fn sandbox_watch_event(sandbox: DriverSandbox) -> WatchSandboxesEvent {
WatchSandboxesEvent {
payload: Some(watch_sandboxes_event::Payload::Sandbox(
crates/openshell-server/src/grpc/mod.rs
+17 -10
View File
@@ -48,16 +48,16 @@ use openshell_core::proto::{
ListWorkspacesResponse, MemoryResourceCapabilities, ProviderProfileResponse, ProviderResponse,
PushSandboxLogsRequest, PushSandboxLogsResponse, RefreshSandboxTokenRequest,
RefreshSandboxTokenResponse, RejectDraftChunkRequest, RejectDraftChunkResponse, RelayFrame,
RemoveWorkspaceMemberRequest, RemoveWorkspaceMemberResponse, ReportMainProcessExitRequest,
ReportMainProcessExitResponse, ReportPolicyStatusRequest, ReportPolicyStatusResponse,
ResourceCapabilities, RevokeSshSessionRequest, RevokeSshSessionResponse,
RotateProviderCredentialRequest, RotateProviderCredentialResponse, SandboxResponse,
SandboxTemplateResponse, ServiceEndpointResponse, ServiceStatus, StartSandboxRequest,
StopSandboxRequest, SubmitPolicyAnalysisRequest, SubmitPolicyAnalysisResponse,
SupervisorMessage, TcpForwardFrame, UndoDraftChunkRequest, UndoDraftChunkResponse,
UpdateConfigRequest, UpdateConfigResponse, UpdateProviderProfilesRequest,
UpdateProviderProfilesResponse, UpdateProviderRequest, WatchSandboxRequest,
open_shell_server::OpenShell,
RemoveWorkspaceMemberRequest, RemoveWorkspaceMemberResponse, ReportEndpointStatusRequest,
ReportEndpointStatusResponse, ReportMainProcessExitRequest, ReportMainProcessExitResponse,
ReportPolicyStatusRequest, ReportPolicyStatusResponse, ResourceCapabilities,
RevokeSshSessionRequest, RevokeSshSessionResponse, RotateProviderCredentialRequest,
RotateProviderCredentialResponse, SandboxResponse, SandboxTemplateResponse,
ServiceEndpointResponse, ServiceStatus, StartSandboxRequest, StopSandboxRequest,
SubmitPolicyAnalysisRequest, SubmitPolicyAnalysisResponse, SupervisorMessage, TcpForwardFrame,
UndoDraftChunkRequest, UndoDraftChunkResponse, UpdateConfigRequest, UpdateConfigResponse,
UpdateProviderProfilesRequest, UpdateProviderProfilesResponse, UpdateProviderRequest,
WatchSandboxRequest, open_shell_server::OpenShell,
};
use serde::{Deserialize, Serialize};
use std::collections::BTreeMap;
@@ -592,6 +592,13 @@ impl OpenShell for OpenShellService {
policy::handle_report_policy_status(&self.state, request).await
}
async fn report_endpoint_status(
&self,
request: Request<ReportEndpointStatusRequest>,
) -> Result<Response<ReportEndpointStatusResponse>, Status> {
policy::handle_report_endpoint_status(&self.state, request).await
}
// --- Sandbox logs ---
async fn get_sandbox_logs(
crates/openshell-server/src/grpc/policy.rs
+80 -204
View File
@@ -10,6 +10,14 @@
#![allow(clippy::cast_precision_loss)] // f64->f32 for confidence scores
#![allow(clippy::items_after_statements)] // DB_PORTS const inside function
mod endpoint_status;
pub(super) use endpoint_status::handle_report_endpoint_status;
pub use endpoint_status::{
invalidate_endpoint_status_on_startup, reset_endpoint_status_for_supervisor_session,
retry_endpoint_status_after_supervisor_disconnect,
};
use crate::ServerState;
use crate::auth::principal::Principal;
use crate::auth::workspace_authz::{
@@ -30,6 +38,8 @@ use crate::storage_proto::StoredProviderCredentialRefreshState;
use crate::storage_proto::StoredProviderProfile;
use metrics::counter;
use openshell_core::net::{is_always_blocked_ip, is_internal_ip};
use openshell_core::policy_identity::canonical_rule_bytes;
pub use openshell_core::policy_identity::deterministic_policy_hash;
#[cfg(test)]
use openshell_core::proto::StaticCredentialBinding;
use openshell_core::proto::policy_merge_operation;
@@ -3075,6 +3085,7 @@ async fn profile_provider_policy_layers_with_catalog(
pub(super) struct CredentialedEndpointScope {
host: String,
ports: Vec<u32>,
path: String,
}
#[derive(Debug, Default)]
@@ -3131,6 +3142,7 @@ async fn provider_policy_context_with_catalog(
let scope = CredentialedEndpointScope {
host: endpoint.host.to_ascii_lowercase(),
ports: endpoint_ports(endpoint),
path: canonical_endpoint_path(&endpoint.path),
};
if !credentialed_scopes.contains(&scope) {
credentialed_scopes.push(scope);
@@ -3174,6 +3186,7 @@ fn extend_credentialed_scopes_from_policy_bindings(
let scope = CredentialedEndpointScope {
host: binding.host.to_ascii_lowercase(),
ports: vec![binding.port],
path: canonical_endpoint_path(&binding.path),
};
if !scopes.contains(&scope) {
scopes.push(scope);
@@ -3190,9 +3203,17 @@ fn endpoint_matches_credentialed_scope(
return false;
}
let endpoint_ports = endpoint_ports(endpoint);
endpoint_ports.is_empty()
let port_matches = endpoint_ports.is_empty()
|| scope.ports.is_empty()
|| endpoint_ports.iter().any(|port| scope.ports.contains(port))
|| endpoint_ports.iter().any(|port| scope.ports.contains(port));
port_matches && path_pattern_covers(&scope.path, &canonical_endpoint_path(&endpoint.path))
}
fn canonical_endpoint_path(path: &str) -> String {
match path.trim() {
"" | "**" | "/**" => "/**".to_string(),
path => path.to_string(),
}
}
pub(super) fn clear_provider_credentialed_markers(policy: &mut ProtoSandboxPolicy) {
@@ -3640,6 +3661,10 @@ async fn handle_update_config_inner(
let payload = new_policy.encode_to_vec();
let hash = deterministic_policy_hash(&new_policy);
// Global policy determines the report's effective configuration.
// Serialize its writes after validation so a report cannot commit
// evidence derived from the policy this update has replaced.
let _sandbox_sync_guard = state.compute.sandbox_sync_guard().await;
let latest = state
.store
.get_latest_policy(GLOBAL_POLICY_SANDBOX_ID)
@@ -3745,6 +3770,13 @@ async fn handle_update_config_inner(
validate_registered_setting_key(key)?;
}
// Deleting global policy changes the report's effective configuration.
// Keep settings -> sandbox lock order for all global policy mutations.
let _sandbox_sync_guard = if key == POLICY_SETTING_KEY && req.delete_setting {
Some(state.compute.sandbox_sync_guard().await)
} else {
None
};
let mut global_settings = load_global_settings(state.store.as_ref()).await?;
let provider_composition_was_enabled =
provider_policy_composition_enabled_in(&global_settings)?;
@@ -4511,17 +4543,38 @@ pub async fn record_policy_apply_result(
.supersede_older_policies(sandbox_id, version_i64)
.await;
let _sandbox_sync_guard = state.compute.sandbox_sync_guard().await;
state
let sandbox = state
.store
.get_message::<Sandbox>(sandbox_id)
.await
.map_err(|error| Status::internal(format!("fetch sandbox failed: {error}")))?
.ok_or_else(|| Status::not_found("sandbox not found"))?;
let endpoint_reset = endpoint_status::endpoint_status_reset_for_loaded_policy(
state.as_ref(),
&sandbox,
version_i64,
)
.await?;
let updated = state
.store
.update_message_cas::<Sandbox, _>(sandbox_id, 0, |sandbox| {
if sandbox.current_policy_version() < version {
sandbox.set_current_policy_version(version);
}
if let Some(endpoints) = &endpoint_reset {
endpoint_status::reconcile_endpoint_statuses(
sandbox,
endpoints,
&HashSet::new(),
"",
);
}
})
.await
.map_err(|error| {
super::persistence_error_to_status(error, "update current_policy_version")
})?;
state.sandbox_index.update_from_sandbox(&updated);
state.sandbox_watch_bus.notify(sandbox_id);
}
Ok(())
@@ -6058,203 +6111,6 @@ pub(super) async fn handle_get_draft_history(
// Policy helper functions
// ---------------------------------------------------------------------------
fn append_canonical_bytes(out: &mut Vec<u8>, value: &[u8]) {
out.extend_from_slice(
&u64::try_from(value.len())
.expect("canonical value length fits in u64")
.to_le_bytes(),
);
out.extend_from_slice(value);
}
fn append_canonical_message<M: Message>(out: &mut Vec<u8>, value: &M) {
append_canonical_bytes(out, &value.encode_to_vec());
}
fn append_sorted_message_map<M: Message>(
out: &mut Vec<u8>,
label: &[u8],
values: &HashMap<String, M>,
) {
append_canonical_bytes(out, label);
let mut entries = values.iter().collect::<Vec<_>>();
entries.sort_by_key(|(key, _)| key.as_str());
out.extend_from_slice(
&u64::try_from(entries.len())
.expect("canonical map length fits in u64")
.to_le_bytes(),
);
for (key, value) in entries {
append_canonical_bytes(out, key.as_bytes());
append_canonical_message(out, value);
}
}
/// Encode a policy rule without depending on randomized protobuf map order.
fn canonical_rule_bytes(rule: &NetworkPolicyRule) -> Vec<u8> {
let mut map_free = rule.clone();
for endpoint in &mut map_free.endpoints {
endpoint.graphql_persisted_queries.clear();
for rule in &mut endpoint.rules {
if let Some(allow) = &mut rule.allow {
allow.query.clear();
allow.params.clear();
}
}
for deny in &mut endpoint.deny_rules {
deny.query.clear();
deny.params.clear();
}
}
let mut out = Vec::new();
append_canonical_message(&mut out, &map_free);
for (endpoint_index, endpoint) in rule.endpoints.iter().enumerate() {
out.extend_from_slice(
&u64::try_from(endpoint_index)
.expect("endpoint index fits in u64")
.to_le_bytes(),
);
append_sorted_message_map(
&mut out,
b"graphql_persisted_queries",
&endpoint.graphql_persisted_queries,
);
for (rule_index, rule) in endpoint.rules.iter().enumerate() {
out.extend_from_slice(
&u64::try_from(rule_index)
.expect("rule index fits in u64")
.to_le_bytes(),
);
if let Some(allow) = &rule.allow {
append_sorted_message_map(&mut out, b"allow_query", &allow.query);
append_sorted_message_map(&mut out, b"allow_params", &allow.params);
}
}
for (rule_index, deny) in endpoint.deny_rules.iter().enumerate() {
out.extend_from_slice(
&u64::try_from(rule_index)
.expect("deny-rule index fits in u64")
.to_le_bytes(),
);
append_sorted_message_map(&mut out, b"deny_query", &deny.query);
append_sorted_message_map(&mut out, b"deny_params", &deny.params);
}
}
out
}
fn canonical_struct_bytes(value: &prost_types::Struct) -> Vec<u8> {
let mut out = Vec::new();
let mut fields = value.fields.iter().collect::<Vec<_>>();
fields.sort_by_key(|(key, _)| key.as_str());
out.extend_from_slice(
&u64::try_from(fields.len())
.expect("struct field count fits in u64")
.to_le_bytes(),
);
for (key, value) in fields {
append_canonical_bytes(&mut out, key.as_bytes());
append_canonical_bytes(&mut out, &canonical_value_bytes(value));
}
out
}
fn canonical_value_bytes(value: &prost_types::Value) -> Vec<u8> {
use prost_types::value::Kind;
let mut out = Vec::new();
match &value.kind {
None => out.push(0),
Some(Kind::NullValue(value)) => {
out.push(1);
out.extend_from_slice(&value.to_le_bytes());
}
Some(Kind::NumberValue(value)) => {
out.push(2);
out.extend_from_slice(&value.to_bits().to_le_bytes());
}
Some(Kind::StringValue(value)) => {
out.push(3);
append_canonical_bytes(&mut out, value.as_bytes());
}
Some(Kind::BoolValue(value)) => {
out.push(4);
out.push(u8::from(*value));
}
Some(Kind::StructValue(value)) => {
out.push(5);
append_canonical_bytes(&mut out, &canonical_struct_bytes(value));
}
Some(Kind::ListValue(value)) => {
out.push(6);
out.extend_from_slice(
&u64::try_from(value.values.len())
.expect("list length fits in u64")
.to_le_bytes(),
);
for item in &value.values {
append_canonical_bytes(&mut out, &canonical_value_bytes(item));
}
}
}
out
}
fn canonical_middleware_bytes(
middleware: &openshell_core::proto::NetworkMiddlewareConfig,
) -> Vec<u8> {
let mut map_free = middleware.clone();
map_free.config = None;
let mut out = Vec::new();
append_canonical_message(&mut out, &map_free);
if let Some(config) = &middleware.config {
append_canonical_bytes(&mut out, &canonical_struct_bytes(config));
}
out
}
fn canonical_policy_bytes(policy: &ProtoSandboxPolicy) -> Vec<u8> {
let mut map_free = policy.clone();
map_free.network_policies.clear();
map_free.network_middlewares.clear();
let mut out = Vec::new();
append_canonical_message(&mut out, &map_free);
let mut policy_entries = policy.network_policies.iter().collect::<Vec<_>>();
policy_entries.sort_by_key(|(key, _)| key.as_str());
append_canonical_bytes(&mut out, b"network_policies");
out.extend_from_slice(
&u64::try_from(policy_entries.len())
.expect("policy count fits in u64")
.to_le_bytes(),
);
for (key, rule) in policy_entries {
append_canonical_bytes(&mut out, key.as_bytes());
append_canonical_bytes(&mut out, &canonical_rule_bytes(rule));
}
let mut middleware_entries = policy.network_middlewares.iter().collect::<Vec<_>>();
middleware_entries.sort_by_key(|(key, _)| key.as_str());
append_canonical_bytes(&mut out, b"network_middlewares");
out.extend_from_slice(
&u64::try_from(middleware_entries.len())
.expect("middleware count fits in u64")
.to_le_bytes(),
);
for (key, middleware) in middleware_entries {
append_canonical_bytes(&mut out, key.as_bytes());
append_canonical_bytes(&mut out, &canonical_middleware_bytes(middleware));
}
out
}
/// Compute a deterministic SHA-256 hash of a `SandboxPolicy`, recursively
/// sorting every protobuf map while preserving repeated-field order.
pub fn deterministic_policy_hash(policy: &ProtoSandboxPolicy) -> String {
hex::encode(Sha256::digest(canonical_policy_bytes(policy)))
}
/// Rebuilds a policy revision's identity from its checked canonical payload.
///
/// `PolicyRecord::policy_hash` is persisted metadata and cannot prove what the
@@ -7712,7 +7568,7 @@ mod tests {
/// Wrap a request with a sandbox `Principal` bound to `sandbox_id`.
/// Use for tests that exercise sandbox-caller code paths.
#[allow(dead_code)]
fn with_sandbox<T>(mut request: Request<T>, sandbox_id: &str) -> Request<T> {
pub(super) fn with_sandbox<T>(mut request: Request<T>, sandbox_id: &str) -> Request<T> {
request
.extensions_mut()
.insert(Principal::Sandbox(SandboxPrincipal {
@@ -7736,7 +7592,7 @@ mod tests {
})
}
fn mcp_policy_with_versions(versions: &[&str]) -> ProtoSandboxPolicy {
pub(super) fn mcp_policy_with_versions(versions: &[&str]) -> ProtoSandboxPolicy {
let mut policy = openshell_policy::restrictive_default_policy();
policy.network_policies.insert(
"mcp".to_string(),
@@ -8833,7 +8689,7 @@ mod tests {
}
#[test]
fn provider_credentialed_stamping_matches_host_patterns_and_ports() {
fn provider_credentialed_stamping_matches_host_ports_and_paths() {
let mut policy = ProtoSandboxPolicy {
network_policies: HashMap::from([(
"test".to_string(),
@@ -8842,18 +8698,28 @@ mod tests {
NetworkEndpoint {
host: "api.example.com".to_string(),
port: 443,
path: "/mcp/private/**".to_string(),
provider_credentialed: true,
..Default::default()
},
NetworkEndpoint {
host: "api.example.com".to_string(),
port: 8443,
path: "/mcp/private/**".to_string(),
provider_credentialed: true,
..Default::default()
},
NetworkEndpoint {
host: "*.api.example.com".to_string(),
port: 443,
path: "/mcp/private/**".to_string(),
provider_credentialed: true,
..Default::default()
},
NetworkEndpoint {
host: "api.example.com".to_string(),
port: 443,
path: "/mcp/public/**".to_string(),
provider_credentialed: true,
..Default::default()
},
@@ -8866,6 +8732,7 @@ mod tests {
let scopes = vec![CredentialedEndpointScope {
host: "*.example.com".to_string(),
ports: vec![443],
path: "/mcp/private/**".to_string(),
}];
clear_provider_credentialed_markers(&mut policy);
@@ -8875,6 +8742,7 @@ mod tests {
assert!(endpoints[0].provider_credentialed);
assert!(!endpoints[1].provider_credentialed);
assert!(!endpoints[2].provider_credentialed);
assert!(!endpoints[3].provider_credentialed);
}
#[test]
@@ -8882,6 +8750,7 @@ mod tests {
let mut scopes = vec![CredentialedEndpointScope {
host: "profile.example.com".to_string(),
ports: vec![443],
path: "/**".to_string(),
}];
let bindings = HashMap::from([
(
@@ -8929,10 +8798,17 @@ mod tests {
CredentialedEndpointScope {
host: "profile.example.com".to_string(),
ports: vec![443],
path: "/**".to_string(),
},
CredentialedEndpointScope {
host: "api.bound.example".to_string(),
ports: vec![8443],
path: "/v1".to_string(),
},
CredentialedEndpointScope {
host: "api.bound.example".to_string(),
ports: vec![8443],
path: "/v2".to_string(),
},
]
);
@@ -9942,7 +9818,7 @@ mod tests {
left
}
fn test_sandbox(
pub(super) fn test_sandbox(
id: &str,
name: &str,
policy: ProtoSandboxPolicy,
crates/openshell-server/src/grpc/policy/endpoint_status.rs
+758
View File
@@ -0,0 +1,758 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
//! Passive network results for configured tool server endpoints.
use super::{
apply_effective_policy_context, canonical_policy_record_identity,
compute_provider_env_revision_with_catalog_and_policy_bindings,
current_effective_policy_for_sandbox, decode_policy_from_global_settings,
deterministic_policy_hash, load_global_settings, policy_static_credential_endpoint_bindings,
};
use crate::ServerState;
use crate::persistence::{ObjectId, ObjectName, ObjectWorkspace};
use crate::policy_store::PolicyStoreExt;
use crate::provider_profile_sources::EffectiveProviderProfileCatalog;
use crate::supervisor_session::EndpointReportCursor;
use chrono::{SecondsFormat, Utc};
use openshell_core::GetResourceVersion;
use openshell_core::endpoint_status::initial_endpoint_status;
use openshell_core::mcp::is_mcp_protocol;
use openshell_core::proto::{
EndpointResult, EndpointStatus, PolicySource, ReportEndpointStatusRequest,
ReportEndpointStatusResponse, Sandbox, SandboxPolicy as ProtoSandboxPolicy, SandboxStatus,
};
use prost::Message;
use sha2::{Digest, Sha256};
use std::collections::{BTreeMap, HashMap, HashSet};
use std::sync::Arc;
use tonic::{Request, Response, Status};
use tracing::warn;
const ENDPOINT_STARTUP_RECONCILIATION_PAGE_SIZE: u32 = 100;
const ENDPOINT_DISCONNECT_RETRY_INITIAL_BACKOFF: std::time::Duration =
std::time::Duration::from_millis(100);
const ENDPOINT_DISCONNECT_RETRY_MAX_BACKOFF: std::time::Duration =
std::time::Duration::from_secs(5);
#[derive(Clone, Debug, PartialEq)]
struct ExpectedEndpoint {
// Credential attribution is derived from the effective policy so a
// sandbox cannot claim credential failure for an unauthenticated endpoint.
provider_credentialed: bool,
// The public status comes only from the validated policy endpoint,
// never from observed request data or supervisor-supplied text.
status: EndpointStatus,
}
struct EndpointContext {
policy_hash: String,
provider_env_revision: u64,
endpoints: BTreeMap<String, ExpectedEndpoint>,
}
/// Accept the last observed result for each configured tool server endpoint.
///
/// Addresses and identifiers come from the authoritative policy. The supervisor
/// supplies only typed results, so request data and upstream text cannot enter status.
pub(in crate::grpc) async fn handle_report_endpoint_status(
state: &Arc<ServerState>,
request: Request<ReportEndpointStatusRequest>,
) -> Result<Response<ReportEndpointStatusResponse>, Status> {
let sandbox_id = request.get_ref().sandbox_id.clone();
crate::auth::guard::enforce_sandbox_scope(&request, &sandbox_id)?;
let req = request.into_inner();
if req.sandbox_id.is_empty() {
return Err(Status::invalid_argument("sandbox_id is required"));
}
if req.policy_hash.is_empty() {
return Err(Status::invalid_argument("policy_hash is required"));
}
if req.supervisor_session_id.is_empty() {
return Err(Status::invalid_argument(
"supervisor_session_id is required",
));
}
if req.report_sequence == 0 {
return Err(Status::invalid_argument("report_sequence is required"));
}
// Session validation, configuration derivation, and persistence share the
// sandbox mutation boundary. A newly registered supervisor can therefore
// invalidate its predecessor before any stale report reaches the CAS.
let _sandbox_sync_guard = state.compute.sandbox_sync_guard().await;
if !state
.supervisor_sessions
.is_endpoint_status_authority(&req.sandbox_id, &req.supervisor_session_id)
{
return Err(Status::permission_denied(
"tool server endpoint status requires the active supervisor session",
));
}
let sandbox = state
.store
.get_message::<Sandbox>(&req.sandbox_id)
.await
.map_err(|error| Status::internal(format!("fetch sandbox failed: {error}")))?
.ok_or_else(|| Status::not_found("sandbox not found"))?;
let context = active_endpoint_context(state.as_ref(), &sandbox).await?;
if req.policy_hash != context.policy_hash
|| req.provider_env_revision != context.provider_env_revision
{
return Err(Status::failed_precondition(
"tool server endpoint status revisions do not match the current sandbox configuration",
));
}
let current_cursor = state
.supervisor_sessions
.endpoint_report_cursor(&req.sandbox_id, &req.supervisor_session_id);
let report_digest: [u8; 32] = Sha256::digest(req.encode_to_vec()).into();
if let Some(cursor) = current_cursor.as_ref() {
if req.report_sequence < cursor.report_sequence {
return Err(Status::failed_precondition(
"tool server endpoint report sequence is older than the current session cursor",
));
}
if req.report_sequence == cursor.report_sequence {
if cursor.policy_hash != req.policy_hash
|| cursor.provider_env_revision != req.provider_env_revision
|| cursor.report_digest != report_digest
{
return Err(Status::invalid_argument(
"a tool server endpoint report sequence must reuse the identical request",
));
}
// The prior write committed but its acknowledgement was lost. Do
// not fabricate observation times by reconciling the same batch.
return Ok(Response::new(ReportEndpointStatusResponse {}));
}
}
// Sequences belong to the authenticated session, not configuration values.
// Gaps represent snapshots superseded by an inventory reset; accepting a
// newer complete snapshot retires every lower sequence, including an RPC
// cancelled after dispatch or a report from an earlier equal-valued epoch.
let (reports, observed_endpoint_ids) = validate_endpoint_snapshot(
&req.observations,
&req.observed_endpoint_ids,
&context.endpoints,
)?;
validate_endpoint_observation_markers(&sandbox, &reports, &observed_endpoint_ids)?;
let now = Utc::now().to_rfc3339_opts(SecondsFormat::Millis, true);
let expected_resource_version = sandbox.get_resource_version();
let updated = state
.store
.update_message_cas::<Sandbox, _>(&req.sandbox_id, expected_resource_version, |sandbox| {
reconcile_endpoint_statuses(sandbox, &reports, &observed_endpoint_ids, &now);
})
.await
.map_err(|error| {
super::super::persistence_error_to_status(error, "persist tool server endpoint status")
})?;
state.sandbox_index.update_from_sandbox(&updated);
state.sandbox_watch_bus.notify(&req.sandbox_id);
if !state.supervisor_sessions.commit_endpoint_report_cursor(
&req.sandbox_id,
&req.supervisor_session_id,
EndpointReportCursor {
policy_hash: req.policy_hash,
provider_env_revision: req.provider_env_revision,
report_sequence: req.report_sequence,
report_digest,
},
) {
return Err(Status::permission_denied(
"tool server endpoint status supervisor session was replaced while persisting the report",
));
}
Ok(Response::new(ReportEndpointStatusResponse {}))
}
async fn current_endpoint_context(
state: &ServerState,
sandbox: &Sandbox,
) -> Result<EndpointContext, Status> {
let workspace = sandbox.object_workspace();
let catalog = state
.provider_profile_sources
.snapshot_catalog(state.store.as_ref(), workspace)
.await?;
let policy = current_effective_policy_for_sandbox(
state,
&catalog,
workspace,
sandbox,
sandbox.object_id(),
)
.await?;
derive_endpoint_context(state, sandbox, &catalog, policy).await
}
async fn active_endpoint_context(
state: &ServerState,
sandbox: &Sandbox,
) -> Result<EndpointContext, Status> {
let version = sandbox.current_policy_version();
if version == 0 {
// Before the first load acknowledgement there is no active revision.
// The effective policy is the candidate sent to the supervisor, so it
// is the only configuration against which the initial session can be
// reset and subsequently report.
return current_endpoint_context(state, sandbox).await;
}
// A newer stored policy may still be pending while the supervisor runs the
// acknowledged revision. Endpoint observations belong to that active runtime
// configuration until another load acknowledgement commits the transition.
endpoint_context_for_loaded_policy(state, sandbox, i64::from(version)).await
}
async fn endpoint_context_for_loaded_policy(
state: &ServerState,
sandbox: &Sandbox,
version: i64,
) -> Result<EndpointContext, Status> {
let workspace = sandbox.object_workspace();
let catalog = state
.provider_profile_sources
.snapshot_catalog(state.store.as_ref(), workspace)
.await?;
let global_settings = load_global_settings(state.store.as_ref()).await?;
let global_policy = decode_policy_from_global_settings(&global_settings)?;
endpoint_context_for_loaded_policy_with_inputs(
state,
sandbox,
version,
&catalog,
global_policy.as_ref(),
)
.await
}
async fn endpoint_context_for_loaded_policy_with_inputs(
state: &ServerState,
sandbox: &Sandbox,
version: i64,
catalog: &EffectiveProviderProfileCatalog,
global_policy: Option<&ProtoSandboxPolicy>,
) -> Result<EndpointContext, Status> {
let workspace = sandbox.object_workspace();
let provider_names = sandbox
.spec
.as_ref()
.map(|spec| spec.providers.clone())
.unwrap_or_default();
let policy = if let Some(global_policy) = global_policy {
apply_effective_policy_context(
state,
catalog,
workspace,
&provider_names,
global_policy.clone(),
PolicySource::Global,
)
.await?
} else {
let record = state
.store
.get_policy_by_version(sandbox.object_id(), version)
.await
.map_err(|error| Status::internal(format!("fetch policy revision failed: {error}")))?
.ok_or_else(|| Status::not_found("policy revision not found"))?;
let policy = canonical_policy_record_identity(&record)?.0;
apply_effective_policy_context(
state,
catalog,
workspace,
&provider_names,
policy,
PolicySource::Sandbox,
)
.await?
};
derive_endpoint_context(state, sandbox, catalog, policy).await
}
async fn derive_endpoint_context(
state: &ServerState,
sandbox: &Sandbox,
catalog: &EffectiveProviderProfileCatalog,
policy: ProtoSandboxPolicy,
) -> Result<EndpointContext, Status> {
let workspace = sandbox.object_workspace();
let provider_names = sandbox
.spec
.as_ref()
.map(|spec| spec.providers.clone())
.unwrap_or_default();
let bindings = policy_static_credential_endpoint_bindings(Some(&policy))?;
let provider_env_revision = compute_provider_env_revision_with_catalog_and_policy_bindings(
state.store.as_ref(),
catalog,
workspace,
&provider_names,
&bindings,
)
.await?;
Ok(EndpointContext {
policy_hash: deterministic_policy_hash(&policy),
provider_env_revision,
endpoints: expected_endpoint_statuses(&policy),
})
}
fn unknown_endpoint_reports(
endpoints: &BTreeMap<String, ExpectedEndpoint>,
) -> BTreeMap<String, EndpointStatus> {
endpoints
.iter()
.map(|(endpoint_id, endpoint)| {
(
endpoint_id.clone(),
EndpointStatus {
last_result: EndpointResult::NoObservedExchange as i32,
..endpoint.status.clone()
},
)
})
.collect()
}
/// Reset endpoint evidence before acknowledging a newly authenticated supervisor.
pub async fn reset_endpoint_status_for_supervisor_session(
state: &Arc<ServerState>,
sandbox_id: &str,
supervisor_session_id: &str,
) -> Result<(), Status> {
let _sandbox_sync_guard = state.compute.sandbox_sync_guard().await;
if !state
.supervisor_sessions
.is_current_session(sandbox_id, supervisor_session_id)
{
return Err(Status::failed_precondition(
"supervisor session was replaced before endpoint status reset",
));
}
let sandbox = state
.store
.get_message::<Sandbox>(sandbox_id)
.await
.map_err(|error| Status::internal(format!("fetch sandbox failed: {error}")))?
.ok_or_else(|| Status::not_found("sandbox not found"))?;
let context = active_endpoint_context(state.as_ref(), &sandbox).await?;
let reports = unknown_endpoint_reports(&context.endpoints);
let now = Utc::now().to_rfc3339_opts(SecondsFormat::Millis, true);
let expected_resource_version = sandbox.get_resource_version();
let updated = state
.store
.update_message_cas::<Sandbox, _>(sandbox_id, expected_resource_version, |sandbox| {
reconcile_endpoint_statuses(sandbox, &reports, &HashSet::new(), &now);
})
.await
.map_err(|error| {
super::super::persistence_error_to_status(
error,
"reset tool server endpoint status for supervisor",
)
})?;
state.sandbox_index.update_from_sandbox(&updated);
state.sandbox_watch_bus.notify(sandbox_id);
Ok(())
}
/// Reset endpoint observations after the active supervisor stream disconnects.
///
/// A concurrently registered replacement owns its own pre-acknowledgement
/// reset, so this path leaves that session's cursor alone.
pub async fn reset_endpoint_status_after_supervisor_disconnect(
state: &Arc<ServerState>,
sandbox_id: &str,
) -> Result<(), Status> {
let _sandbox_sync_guard = state.compute.sandbox_sync_guard().await;
if state
.supervisor_sessions
.current_session_id(sandbox_id)
.is_some()
{
return Ok(());
}
let sandbox = state
.store
.get_message::<Sandbox>(sandbox_id)
.await
.map_err(|error| Status::internal(format!("fetch sandbox failed: {error}")))?
.ok_or_else(|| Status::not_found("sandbox not found"))?;
let context = active_endpoint_context(state.as_ref(), &sandbox).await?;
let reports = unknown_endpoint_reports(&context.endpoints);
let now = Utc::now().to_rfc3339_opts(SecondsFormat::Millis, true);
let expected_resource_version = sandbox.get_resource_version();
let updated = state
.store
.update_message_cas::<Sandbox, _>(sandbox_id, expected_resource_version, |sandbox| {
reconcile_endpoint_statuses(sandbox, &reports, &HashSet::new(), &now);
})
.await
.map_err(|error| {
super::super::persistence_error_to_status(
error,
"reset tool server endpoint status after disconnect",
)
})?;
state.sandbox_index.update_from_sandbox(&updated);
state.sandbox_watch_bus.notify(sandbox_id);
Ok(())
}
/// Retry disconnect invalidation until it is durable or a replacement session
/// takes ownership.
///
/// A transient storage failure must not leave a previously successful endpoint
/// usable after its observation authority has disappeared. Gateway restart is
/// covered separately by startup reconciliation before listeners are bound.
pub async fn retry_endpoint_status_after_supervisor_disconnect(
state: Arc<ServerState>,
sandbox_id: String,
) {
let mut backoff = ENDPOINT_DISCONNECT_RETRY_INITIAL_BACKOFF;
loop {
match reset_endpoint_status_after_supervisor_disconnect(&state, &sandbox_id).await {
Ok(()) => return,
Err(error) if error.code() == tonic::Code::NotFound => return,
Err(error) => {
warn!(
sandbox_id = %sandbox_id,
error = %error,
retry_after_ms = backoff.as_millis(),
"supervisor session: retrying tool server endpoint-status invalidation"
);
tokio::time::sleep(backoff).await;
backoff = (backoff * 2).min(ENDPOINT_DISCONNECT_RETRY_MAX_BACKOFF);
}
}
}
}
/// Invalidate endpoint results left by sessions from an earlier gateway process.
///
/// Supervisor sessions are intentionally process-local. This reconciliation
/// runs before gateway listeners are bound, so persisted success can never be
/// served without a session in the current process that owns the observation.
pub async fn invalidate_endpoint_status_on_startup(state: &Arc<ServerState>) -> Result<(), Status> {
let mut offset = 0;
loop {
let sandboxes = state
.store
.list_all_messages::<Sandbox>(ENDPOINT_STARTUP_RECONCILIATION_PAGE_SIZE, offset)
.await
.map_err(|error| {
Status::internal(format!(
"list sandboxes for tool server endpoint-status startup reconciliation failed: {error}"
))
})?;
if sandboxes.is_empty() {
return Ok(());
}
for sandbox in &sandboxes {
let has_endpoint_status = sandbox
.status
.as_ref()
.is_some_and(|status| !status.endpoint_statuses.is_empty());
if !has_endpoint_status {
continue;
}
let sandbox_id = sandbox.object_id();
let expected_resource_version = sandbox.get_resource_version();
let updated = state
.store
.update_message_cas::<Sandbox, _>(
sandbox_id,
expected_resource_version,
invalidate_endpoint_status_without_session,
)
.await
.map_err(|error| {
super::super::persistence_error_to_status(
error,
"invalidate tool server endpoint status during gateway startup",
)
})?;
state.sandbox_index.update_from_sandbox(&updated);
}
let page_len = sandboxes.len() as u32;
if page_len < ENDPOINT_STARTUP_RECONCILIATION_PAGE_SIZE {
return Ok(());
}
offset = offset.checked_add(page_len).ok_or_else(|| {
Status::internal(
"sandbox pagination overflow during tool server endpoint-status reconciliation",
)
})?;
}
}
fn invalidate_endpoint_status_without_session(sandbox: &mut Sandbox) {
let Some(status) = sandbox.status.as_mut() else {
return;
};
// Losing observation authority clears evidence, while policy-derived identity
// remains available so callers can still identify the configured endpoint.
for endpoint in &mut status.endpoint_statuses {
endpoint.last_result = EndpointResult::NoObservedExchange as i32;
endpoint.last_reported_at.clear();
}
}
fn expected_endpoint_statuses(policy: &ProtoSandboxPolicy) -> BTreeMap<String, ExpectedEndpoint> {
let mut endpoints = BTreeMap::new();
for endpoint in policy
.network_policies
.values()
.flat_map(|rule| &rule.endpoints)
.filter(|endpoint| is_mcp_protocol(&endpoint.protocol))
{
let status = initial_endpoint_status(endpoint);
endpoints
.entry(status.endpoint_id.clone())
.and_modify(|expected: &mut ExpectedEndpoint| {
expected.provider_credentialed |= endpoint.provider_credentialed;
})
.or_insert(ExpectedEndpoint {
provider_credentialed: endpoint.provider_credentialed,
status,
});
}
endpoints
}
fn validate_endpoint_snapshot(
reported: &[openshell_core::proto::EndpointObservation],
reported_observed_endpoint_ids: &[String],
expected: &BTreeMap<String, ExpectedEndpoint>,
) -> Result<(BTreeMap<String, EndpointStatus>, HashSet<String>), Status> {
let mut validated = BTreeMap::new();
for endpoint in reported {
if endpoint.endpoint_id.is_empty() {
return Err(Status::invalid_argument("endpoint_id is required"));
}
let Some(expected_endpoint) = expected.get(&endpoint.endpoint_id) else {
return Err(Status::failed_precondition(
"tool server endpoint status contains an endpoint_id outside the current policy",
));
};
let result = EndpointResult::try_from(endpoint.result)
.map_err(|_| Status::invalid_argument("tool server endpoint result is invalid"))?;
if result == EndpointResult::Unspecified {
return Err(Status::invalid_argument(
"tool server endpoint result is required",
));
}
if result == EndpointResult::CredentialUnavailable
&& !expected_endpoint.provider_credentialed
{
return Err(Status::invalid_argument(
"credential-unavailable requires a credentialed tool server endpoint",
));
}
if validated
.insert(
endpoint.endpoint_id.clone(),
EndpointStatus {
last_result: result as i32,
..expected_endpoint.status.clone()
},
)
.is_some()
{
return Err(Status::invalid_argument("endpoint_ids must be unique"));
}
}
if validated.len() != expected.len()
|| validated
.keys()
.zip(expected.keys())
.any(|(reported, expected)| reported != expected)
{
return Err(Status::failed_precondition(
"tool server endpoint status must contain the complete current endpoint set",
));
}
let mut observed_endpoint_ids = HashSet::new();
for endpoint_id in reported_observed_endpoint_ids {
let Some(endpoint) = validated.get(endpoint_id) else {
return Err(Status::failed_precondition(
"observed endpoint_id is outside the current snapshot",
));
};
if endpoint.last_result == EndpointResult::NoObservedExchange as i32 {
return Err(Status::invalid_argument(
"an observed tool server endpoint cannot report no observed exchange",
));
}
if !observed_endpoint_ids.insert(endpoint_id.clone()) {
return Err(Status::invalid_argument(
"observed endpoint_ids must be unique",
));
}
}
Ok((validated, observed_endpoint_ids))
}
fn validate_endpoint_observation_markers(
sandbox: &Sandbox,
reports: &BTreeMap<String, EndpointStatus>,
observed_endpoint_ids: &HashSet<String>,
) -> Result<(), Status> {
let previous = sandbox
.status
.as_ref()
.map(|status| {
status
.endpoint_statuses
.iter()
.map(|endpoint| (endpoint.endpoint_id.as_str(), endpoint.last_result))
.collect::<HashMap<_, _>>()
})
.unwrap_or_default();
for (endpoint_id, report) in reports {
if report.last_result == EndpointResult::NoObservedExchange as i32 {
continue;
}
// A retained result may reuse its previous report time. New evidence,
// including evidence accepted after a reset, must carry a marker.
if previous.get(endpoint_id.as_str()) != Some(&report.last_result)
&& !observed_endpoint_ids.contains(endpoint_id)
{
return Err(Status::invalid_argument(
"a new tool server endpoint result must be marked as observed",
));
}
}
Ok(())
}
/// Build an endpoint reset only when the acknowledged policy changes observations' configuration.
/// The caller commits any reset and the active policy version in one CAS.
pub(super) async fn endpoint_status_reset_for_loaded_policy(
state: &ServerState,
sandbox: &Sandbox,
version: i64,
) -> Result<Option<BTreeMap<String, EndpointStatus>>, Status> {
// Both policy versions must use one provider/profile and global-policy view;
// an external catalog can change between independent snapshot requests.
let catalog = state
.provider_profile_sources
.snapshot_catalog(state.store.as_ref(), sandbox.object_workspace())
.await?;
let global_settings = load_global_settings(state.store.as_ref()).await?;
let global_policy = decode_policy_from_global_settings(&global_settings)?;
let context = endpoint_context_for_loaded_policy_with_inputs(
state,
sandbox,
version,
&catalog,
global_policy.as_ref(),
)
.await?;
let current_version = i64::from(sandbox.current_policy_version());
let same_configuration = if current_version == version {
// An acknowledgement can be retried after endpoint reports have already
// committed. Repeating activation must not erase their accepted evidence.
true
} else if current_version != 0 {
// Metadata-only revisions do not reinstall the runtime policy. Compare
// the active policy, not the report cursor: a cursor can outlive an
// intervening policy reset and cannot prove an A -> B -> A transition.
let current = endpoint_context_for_loaded_policy_with_inputs(
state,
sandbox,
current_version,
&catalog,
global_policy.as_ref(),
)
.await?;
current.policy_hash == context.policy_hash
} else {
// The first endpoint report and load acknowledgement travel independently.
// Only the current session's accepted report proves which configuration
// produced evidence before any loaded policy version has been recorded.
state
.supervisor_sessions
.current_session_id(sandbox.object_id())
.and_then(|session_id| {
state
.supervisor_sessions
.endpoint_report_cursor(sandbox.object_id(), &session_id)
})
.is_some_and(|cursor| {
cursor.policy_hash == context.policy_hash
&& cursor.provider_env_revision == context.provider_env_revision
})
};
Ok((!same_configuration).then(|| unknown_endpoint_reports(&context.endpoints)))
}
/// Replace the endpoint inventory and evidence without changing lifecycle state.
pub(super) fn reconcile_endpoint_statuses(
sandbox: &mut Sandbox,
reports: &BTreeMap<String, EndpointStatus>,
observed_endpoint_ids: &HashSet<String>,
now: &str,
) {
let phase = sandbox.phase();
let current_policy_version = sandbox.current_policy_version();
let sandbox_name = if sandbox.object_name().is_empty() {
sandbox.object_id()
} else {
sandbox.object_name()
}
.to_string();
let status = sandbox.status.get_or_insert_with(|| SandboxStatus {
sandbox_name,
phase,
current_policy_version,
..Default::default()
});
let previous = status
.endpoint_statuses
.iter()
.map(|endpoint| {
(
endpoint.endpoint_id.as_str(),
endpoint.last_reported_at.as_str(),
)
})
.collect::<HashMap<_, _>>();
// Replace the complete inventory to remove retired endpoints atomically.
// Only newly accepted evidence advances the gateway acceptance timestamp;
// unchanged evidence retains its timestamp and unknown results have none.
status.endpoint_statuses = reports
.iter()
.map(|(endpoint_id, report)| {
let mut endpoint = report.clone();
endpoint.last_reported_at =
if endpoint.last_result == EndpointResult::NoObservedExchange as i32 {
String::new()
} else if observed_endpoint_ids.contains(endpoint_id) {
now.to_string()
} else {
previous
.get(endpoint_id.as_str())
.copied()
.unwrap_or_default()
.to_string()
};
endpoint
})
.collect();
}
#[cfg(test)]
#[path = "endpoint_status_tests.rs"]
mod tests;
crates/openshell-server/src/grpc/policy/endpoint_status_tests.rs
+1231
View File
File diff suppressed because it is too large Load Diff
crates/openshell-server/src/grpc/provider.rs
+4
View File
@@ -3741,6 +3741,10 @@ pub(super) async fn handle_update_provider(
let workspace = super::workspace::resolve_workspace(state.store.as_ref(), &authz.workspace)
.await?
.name;
// Provider material contributes to the route-report configuration epoch.
// Serialize its mutation with route-status validation so a report derived
// from the prior revision cannot commit after this update.
let _sandbox_sync_guard = state.compute.sandbox_sync_guard().await;
let Some(mut provider) = req.provider else {
emit_provider_lifecycle(
"custom",
crates/openshell-server/src/grpc/sandbox.rs
+142 -9
View File
@@ -516,10 +516,13 @@ async fn handle_create_sandbox_inner(
None => None,
};
let sandbox = state
let mut sandbox = state
.compute
.create_sandbox(sandbox, sandbox_token, await_main_process_attachment)
.await?;
state
.supervisor_sessions
.project_endpoint_status(&mut sandbox);
info!(
sandbox_id = %id,
@@ -687,7 +690,10 @@ pub(super) async fn handle_get_sandbox(
.await
.map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))?;
let sandbox = sandbox.ok_or_else(|| Status::not_found("sandbox not found"))?;
let mut sandbox = sandbox.ok_or_else(|| Status::not_found("sandbox not found"))?;
state
.supervisor_sessions
.project_endpoint_status(&mut sandbox);
Ok(Response::new(SandboxResponse {
sandbox: Some(sandbox),
}))
@@ -738,12 +744,17 @@ pub(super) async fn handle_list_sandboxes(
label_selector: selector,
},
};
let page = state
let mut page = state
.store
.list_message_page::<Sandbox>(query, after.as_ref(), pagination.page_size())
.await
.map_err(|e| Status::internal(format!("list sandboxes failed: {e}")))?;
let next_page_token = pagination.next_object_token(page.next_cursor.as_ref());
// Project only the selected page so observation authority cannot change
// the store's ordering or continuation cursor.
for sandbox in &mut page.messages {
state.supervisor_sessions.project_endpoint_status(sandbox);
}
Ok(Response::new(ListSandboxesResponse {
sandboxes: page.messages,
next_page_token,
@@ -1151,7 +1162,7 @@ pub(super) async fn handle_attach_sandbox_provider(
let attached = Arc::new(AtomicBool::new(false));
let attached_clone = attached.clone();
let sandbox = state
let mut sandbox = state
.store
.update_message_cas::<Sandbox, _>(
&sandbox_id,
@@ -1173,6 +1184,9 @@ pub(super) async fn handle_attach_sandbox_provider(
)
.await
.map_err(|e| super::persistence_error_to_status(e, "attach sandbox provider"))?;
state
.supervisor_sessions
.project_endpoint_status(&mut sandbox);
let attached = attached.load(Ordering::Relaxed);
if attached {
@@ -1258,7 +1272,7 @@ pub(super) async fn handle_detach_sandbox_provider(
let detached = Arc::new(AtomicBool::new(false));
let detached_clone = detached.clone();
let sandbox = state
let mut sandbox = state
.store
.update_message_cas::<Sandbox, _>(
&sandbox_id,
@@ -1280,6 +1294,9 @@ pub(super) async fn handle_detach_sandbox_provider(
)
.await
.map_err(|e| super::persistence_error_to_status(e, "detach sandbox provider"))?;
state
.supervisor_sessions
.project_endpoint_status(&mut sandbox);
let detached = detached.load(Ordering::Relaxed);
if detached {
@@ -1390,7 +1407,10 @@ async fn handle_stop_sandbox_inner(
let workspace = super::workspace::resolve_workspace(state.store.as_ref(), &authz.workspace)
.await?
.name;
let sandbox = state.compute.stop_sandbox(&workspace, &req.name).await?;
let mut sandbox = state.compute.stop_sandbox(&workspace, &req.name).await?;
state
.supervisor_sessions
.project_endpoint_status(&mut sandbox);
info!(sandbox_name = %req.name, "StopSandbox request completed successfully");
Ok(Response::new(SandboxResponse {
sandbox: Some(sandbox),
@@ -1434,7 +1454,10 @@ async fn handle_start_sandbox_inner(
let workspace = super::workspace::resolve_workspace(state.store.as_ref(), &authz.workspace)
.await?
.name;
let sandbox = state.compute.start_sandbox(&workspace, &req.name).await?;
let mut sandbox = state.compute.start_sandbox(&workspace, &req.name).await?;
state
.supervisor_sessions
.project_endpoint_status(&mut sandbox);
info!(sandbox_name = %req.name, "StartSandbox request completed successfully");
Ok(Response::new(SandboxResponse {
sandbox: Some(sandbox),
@@ -1574,7 +1597,10 @@ pub(super) async fn handle_watch_sandbox(
// Re-read the snapshot now that we have subscriptions active.
match state.store.get_message::<Sandbox>(&sandbox_id).await {
Ok(Some(sandbox)) => {
Ok(Some(mut sandbox)) => {
state
.supervisor_sessions
.project_endpoint_status(&mut sandbox);
state.sandbox_index.update_from_sandbox(&sandbox);
let _ = tx
.send(Ok(SandboxStreamEvent {
@@ -1656,7 +1682,10 @@ pub(super) async fn handle_watch_sandbox(
match res {
Ok(()) => {
match state.store.get_message::<Sandbox>(&sandbox_id).await {
Ok(Some(sandbox)) => {
Ok(Some(mut sandbox)) => {
state
.supervisor_sessions
.project_endpoint_status(&mut sandbox);
state.sandbox_index.update_from_sandbox(&sandbox);
if tx.send(Ok(SandboxStreamEvent { payload: Some(openshell_core::proto::sandbox_stream_event::Payload::Sandbox(sandbox.clone()))})).await.is_err() {
return;
@@ -6337,6 +6366,110 @@ mod tests {
);
}
#[tokio::test]
async fn list_sandboxes_projects_endpoint_status_across_pages() {
use openshell_core::proto::{EndpointResult, EndpointStatus};
let state = test_server_state().await;
let endpoint = EndpointStatus {
endpoint_id: "endpoint:v1:tools".to_string(),
host: "tools.example.com".to_string(),
ports: vec![443],
path: "/mcp".to_string(),
last_result: EndpointResult::HttpResponseReceived as i32,
last_reported_at: "2026-09-15T01:00:00.000Z".to_string(),
};
for name in ["alpha", "beta", "gamma"] {
let mut sandbox = test_sandbox(name, Vec::new());
sandbox
.status
.as_mut()
.expect("sandbox status")
.endpoint_statuses = vec![endpoint.clone()];
state.store.put_message(&sandbox).await.unwrap();
}
// Only alpha has a live observation authority; the other persisted
// results must be reset in responses, including on the second page.
let (session_tx, _session_rx) = mpsc::channel(1);
let (shutdown_tx, _shutdown_rx) = oneshot::channel();
state.supervisor_sessions.register(
"sandbox-alpha".to_string(),
"session-alpha".to_string(),
session_tx,
shutdown_tx,
);
assert!(
state
.supervisor_sessions
.initialize_endpoint_status_authority("sandbox-alpha", "session-alpha")
);
let request = ListSandboxesRequest {
page_size: 2,
workspace_scope: Some(openshell_core::proto::workspace_selector("default")),
..Default::default()
};
let first_page = handle_list_sandboxes(&state, authed_request(request.clone()))
.await
.unwrap()
.into_inner();
assert_eq!(
first_page
.sandboxes
.iter()
.map(ObjectName::object_name)
.collect::<Vec<_>>(),
["alpha", "beta"]
);
assert!(!first_page.next_page_token.is_empty());
let second_page = handle_list_sandboxes(
&state,
authed_request(ListSandboxesRequest {
page_token: first_page.next_page_token,
..request
}),
)
.await
.unwrap()
.into_inner();
assert_eq!(second_page.sandboxes.len(), 1);
assert_eq!(second_page.sandboxes[0].object_name(), "gamma");
assert!(second_page.next_page_token.is_empty());
let unknown = EndpointStatus {
last_result: EndpointResult::NoObservedExchange as i32,
last_reported_at: String::new(),
..endpoint.clone()
};
for (sandbox, expected_endpoint) in [
(&first_page.sandboxes[0], &endpoint),
(&first_page.sandboxes[1], &unknown),
(&second_page.sandboxes[0], &unknown),
] {
let status = sandbox.status.as_ref().expect("sandbox status");
assert_eq!(status.phase, SandboxPhase::Ready as i32);
assert_eq!(
status.endpoint_statuses.as_slice(),
std::slice::from_ref(expected_endpoint)
);
}
// Read-time projection must not erase persisted reports.
let persisted = state
.store
.get_message::<Sandbox>("sandbox-gamma")
.await
.unwrap()
.expect("persisted sandbox");
assert_eq!(
persisted
.status
.expect("persisted status")
.endpoint_statuses,
[endpoint]
);
}
#[tokio::test]
async fn sandbox_crud_is_workspace_isolated() {
use crate::persistence::ObjectType;
crates/openshell-server/src/lib.rs
+8
View File
@@ -717,6 +717,14 @@ pub(crate) async fn run_server(
error.message()
))
})?;
grpc::policy::invalidate_endpoint_status_on_startup(&state)
.await
.map_err(|error| {
Error::execution(format!(
"tool server endpoint-status startup reconciliation failed: {}",
error.message()
))
})?;
let gateway_listeners = bind_gateway_listeners(
config.bind_address,
crates/openshell-server/src/provider_refresh.rs
+9
View File
@@ -1117,6 +1117,15 @@ async fn apply_minted_credential(
updated.credential_expires_at_ms.remove(key);
}
}
// Acquire the shared sandbox mutation boundary only around validation and
// persistence, after any remote minting or credential staging. This
// prevents route status from committing against the old provider revision
// after the rotation writes, without holding the guard across network I/O.
let _sandbox_sync_guard = if let Some(compute) = compute {
Some(compute.sandbox_sync_guard().await)
} else {
None
};
if let Err(err) = crate::grpc::provider::validate_provider_update_against_attached_sandboxes(
store, workspace, &updated,
)
crates/openshell-server/src/storage_proto.rs
+34 -9
View File
@@ -161,11 +161,14 @@ mod tests {
const STORAGE_V1_SCHEMA_SHA256: &str =
"a623124c961f3a56af58a4ab148c12985e5efaec1ad23441031f5cd5b073fe22";
const PUBLIC_RPC_SCHEMA_SHA256: &str =
"3ba470c3278d0c7a49b7dde7ee735017604afab5054fb7a95665c71a602121f1";
"d6e92d196ee88c91d13b465b2185bdcfc9858771467b06f43bce8794da84f2c9";
const DURABLE_SCHEMA_SHA256: &str =
"96911a750b59ddd3eae48b4539ce1545cf5357ebe830143889ec975216522100";
"d860e070d9ae78741056a170290e45f617b57847a7ec7542119b6f19e1e7e856";
const PUBLIC_DURABLE_OVERLAP_SHA256: &str =
"0d1b24b78becb0025512f35fe320b0d91d0728d3239eabf78cdf99e599eb6af2";
"5fc036292b1e1e3c68fe361c342a56e886926727cac35115f93579cd63898358";
// A persisted Sandbox without endpoint status retains its lifecycle fields;
// the absent repeated field decodes empty and needs no database rewrite.
const SANDBOX_WITHOUT_ENDPOINT_STATUS: &str = "0a1e0a0a73616e64626f782d6964120773616e64626f783a0764656661756c741a2b0a0773616e64626f782a0d0a05526561647912045472756530023807420d73757065727669736f722d6964";
// Synthetic payloads generated with the public declarations at v0.0.116,
// before their relocation into openshell.storage.v1. Values are deliberately
// non-secret and the ordinary protobuf bytes contain no package names.
@@ -491,14 +494,14 @@ mod tests {
}
}
methods.sort();
assert_eq!(compiled_method_count, 100, "classify every compiled RPC");
assert_eq!(methods.len(), 74, "inventory every public gateway RPC");
assert_eq!(compiled_method_count, 101, "classify every compiled RPC");
assert_eq!(methods.len(), 75, "inventory every public gateway RPC");
assert_eq!(
methods
.iter()
.filter(|method| method.starts_with("openshell.v1.OpenShell/"))
.count(),
74
75
);
assert!(methods.iter().all(|method| !method.contains(".storage.")));
@@ -531,13 +534,13 @@ mod tests {
assert_eq!(
(public_closure.messages.len(), public_closure.enums.len()),
(291, 15)
(295, 16)
);
assert_eq!(
(durable_closure.messages.len(), durable_closure.enums.len()),
(84, 11)
(85, 12)
);
assert_eq!((overlap_messages.len(), overlap_enums.len()), (73, 11));
assert_eq!((overlap_messages.len(), overlap_enums.len()), (74, 12));
assert_eq!(
public_inventory_hash, PUBLIC_RPC_SCHEMA_SHA256,
@@ -557,6 +560,28 @@ mod tests {
hex::decode(encoded).expect("checked-in legacy fixture must be valid hex")
}
#[test]
fn sandbox_payload_without_endpoint_status_decodes() {
use openshell_core::proto::{Sandbox, SandboxPhase};
let sandbox = Sandbox::decode(legacy_bytes(SANDBOX_WITHOUT_ENDPOINT_STATUS).as_slice())
.expect("stored sandbox without endpoint status must decode");
let metadata = sandbox.metadata.expect("sandbox metadata");
assert_eq!(metadata.id, "sandbox-id");
assert_eq!(metadata.name, "sandbox");
assert_eq!(metadata.workspace, "default");
let status = sandbox.status.expect("sandbox status");
assert_eq!(status.sandbox_name, "sandbox");
assert_eq!(status.phase(), SandboxPhase::Ready);
assert_eq!(status.current_policy_version, 7);
assert_eq!(status.main_process_instance_id, "supervisor-id");
assert_eq!(status.exit_code, None);
assert_eq!(status.conditions.len(), 1);
assert_eq!(status.conditions[0].r#type, "Ready");
assert_eq!(status.conditions[0].status, "True");
assert!(status.endpoint_statuses.is_empty());
}
#[test]
fn pre_move_storage_payloads_decode_after_package_relocation() {
let refresh = StoredProviderCredentialRefreshState::decode(
crates/openshell-server/src/supervisor_session.rs
+215 -1
View File
@@ -86,6 +86,15 @@ struct LiveSession {
/// Set after the supervisor confirms that every expected foreground
/// attachment has closed and terminal output delivery is complete.
terminal_delivery_finalized: bool,
/// Becomes true only after the gateway durably resets endpoint status for
/// this session and before it sends `SessionAccepted`.
endpoint_status_initialized: bool,
/// Last tool server endpoint-status batch committed for this authenticated session.
///
/// The cursor is session authority state, not public sandbox status. A
/// gateway restart invalidates every session and startup reconciliation
/// resets any persisted endpoint result before requests are served.
endpoint_report_cursor: Option<EndpointReportCursor>,
#[allow(dead_code)]
connected_at: Instant,
}
@@ -112,6 +121,20 @@ struct InFlightConfigUpdate {
sent_at: Instant,
}
/// Idempotency state for tool server endpoint-status reports from one live supervisor.
#[derive(Clone, Debug, PartialEq, Eq)]
pub(crate) struct EndpointReportCursor {
/// Active effective policy represented by the accepted report sequence.
pub(crate) policy_hash: String,
/// Provider environment revision represented by the accepted sequence.
pub(crate) provider_env_revision: u64,
/// Last accepted sequence in this session; superseded snapshots may leave gaps.
pub(crate) report_sequence: u64,
/// Digest of the accepted request, used to reject a different body that
/// reuses an already committed sequence number.
pub(crate) report_digest: [u8; 32],
}
/// Holds a oneshot sender that will deliver the upgraded relay stream or a
/// target-open failure reported by the supervisor.
type RelayStreamSender = oneshot::Sender<Result<tokio::io::DuplexStream, Status>>;
@@ -290,6 +313,8 @@ impl SupervisorSessionRegistry {
config_sequences: ConfigSequences::default(),
shutdown,
terminal_delivery_finalized: false,
endpoint_status_initialized: false,
endpoint_report_cursor: None,
connected_at: Instant::now(),
},
);
@@ -328,7 +353,7 @@ impl SupervisorSessionRegistry {
/// This guards against the supersede race: an old session's task may
/// finish long after a new session has taken its place. The old task's
/// cleanup must not evict the new registration.
fn remove_if_current(&self, sandbox_id: &str, session_id: &str) -> Option<bool> {
pub(crate) fn remove_if_current(&self, sandbox_id: &str, session_id: &str) -> Option<bool> {
let mut sessions = self.sessions.lock().unwrap();
let is_current = sessions
.get(sandbox_id)
@@ -599,6 +624,107 @@ impl SupervisorSessionRegistry {
.is_some_and(|session| session.session_id == session_id)
}
/// Mark the current session as the observation authority after its
/// public endpoint results have been durably reset.
pub(crate) fn initialize_endpoint_status_authority(
&self,
sandbox_id: &str,
session_id: &str,
) -> bool {
let mut sessions = self.sessions.lock().unwrap();
let Some(session) = sessions
.get_mut(sandbox_id)
.filter(|session| session.session_id == session_id)
else {
return false;
};
session.endpoint_status_initialized = true;
true
}
/// Return whether the named session has completed endpoint status
/// initialization and still owns reporting authority.
pub(crate) fn is_endpoint_status_authority(&self, sandbox_id: &str, session_id: &str) -> bool {
self.sessions
.lock()
.unwrap()
.get(sandbox_id)
.is_some_and(|session| {
session.session_id == session_id && session.endpoint_status_initialized
})
}
/// Fail closed when projecting persisted endpoint status without a live,
/// initialized observation authority in the current gateway process.
pub(crate) fn project_endpoint_status(&self, sandbox: &mut Sandbox) {
let sandbox_id = sandbox.object_id();
let has_authority = self
.sessions
.lock()
.unwrap()
.get(sandbox_id)
.is_some_and(|session| session.endpoint_status_initialized);
if has_authority {
return;
}
let Some(status) = sandbox.status.as_mut() else {
return;
};
// Status without its live observation authority is unknown. Keep the
// configured address so a caller can still identify each endpoint.
for endpoint in &mut status.endpoint_statuses {
endpoint.last_result = openshell_core::proto::EndpointResult::NoObservedExchange as i32;
endpoint.last_reported_at.clear();
}
}
/// Return the active supervisor session identifier for gateway-owned
/// status reconciliation.
pub fn current_session_id(&self, sandbox_id: &str) -> Option<String> {
self.sessions
.lock()
.unwrap()
.get(sandbox_id)
.map(|session| session.session_id.clone())
}
/// Return the endpoint report cursor only when `session_id` still owns the
/// sandbox. Replacement sessions never inherit predecessor sequencing.
pub(crate) fn endpoint_report_cursor(
&self,
sandbox_id: &str,
session_id: &str,
) -> Option<EndpointReportCursor> {
self.sessions
.lock()
.unwrap()
.get(sandbox_id)
.filter(|session| session.session_id == session_id)
.and_then(|session| session.endpoint_report_cursor.clone())
}
/// Record a committed endpoint report for the current session.
///
/// Returns `false` when a reconnect replaced the caller while its storage
/// write was in flight. The replacement performs its own pre-acknowledgment
/// reset, so it remains the sole observation authority.
pub(crate) fn commit_endpoint_report_cursor(
&self,
sandbox_id: &str,
session_id: &str,
cursor: EndpointReportCursor,
) -> bool {
let mut sessions = self.sessions.lock().unwrap();
let Some(session) = sessions
.get_mut(sandbox_id)
.filter(|session| session.session_id == session_id)
else {
return false;
};
session.endpoint_report_cursor = Some(cursor);
true
}
fn pending_channel_ids(&self, sandbox_id: &str) -> Vec<String> {
self.pending_relays
.lock()
@@ -1195,6 +1321,30 @@ pub async fn handle_connect_supervisor(
);
}
// A replacement stream is a new observation authority. Reset its endpoint
// results before acknowledging the session so it cannot inherit evidence
// reported by the superseded stream.
if let Err(error) = crate::grpc::policy::reset_endpoint_status_for_supervisor_session(
state,
&sandbox_id,
&session_id,
)
.await
{
state
.supervisor_sessions
.remove_if_current(&sandbox_id, &session_id);
return Err(error);
}
if !state
.supervisor_sessions
.initialize_endpoint_status_authority(&sandbox_id, &session_id)
{
return Err(Status::failed_precondition(
"supervisor session was replaced during endpoint status initialization",
));
}
if superseded {
state
.supervisor_sessions
@@ -1232,6 +1382,12 @@ pub async fn handle_connect_supervisor(
state_clone
.telemetry
.sandbox_session_disconnected(&sandbox_id_clone);
tokio::spawn(
crate::grpc::policy::retry_endpoint_status_after_supervisor_disconnect(
Arc::clone(&state_clone),
sandbox_id_clone.clone(),
),
);
if let Err(err) = state_clone
.compute
.supervisor_session_disconnected(&sandbox_id_clone, terminal_finalized)
@@ -2500,6 +2656,64 @@ mod tests {
}
}
#[test]
fn endpoint_status_projection_requires_initialized_live_authority() {
use openshell_core::proto::{
EndpointResult, EndpointStatus, SandboxCondition, SandboxStatus,
};
let registry = SupervisorSessionRegistry::new();
let mut sandbox = sandbox_record("sandbox-1", "sandbox-1");
let endpoint = EndpointStatus {
endpoint_id: "endpoint:v1:test".to_string(),
host: "api.example.com".to_string(),
ports: vec![443],
path: "/mcp".to_string(),
last_result: EndpointResult::HttpResponseReceived as i32,
last_reported_at: "2026-09-05T01:01:00.000Z".to_string(),
};
let ready = SandboxCondition {
r#type: "Ready".to_string(),
status: "True".to_string(),
..Default::default()
};
sandbox.status = Some(SandboxStatus {
endpoint_statuses: vec![endpoint.clone()],
conditions: vec![ready.clone()],
..Default::default()
});
let unknown = EndpointStatus {
last_result: EndpointResult::NoObservedExchange as i32,
last_reported_at: String::new(),
..endpoint.clone()
};
let mut without_session = sandbox.clone();
registry.project_endpoint_status(&mut without_session);
let projected_status = without_session.status.expect("status");
assert_eq!(projected_status.endpoint_statuses, vec![unknown.clone()]);
assert_eq!(projected_status.conditions, vec![ready.clone()]);
let (session_tx, _session_rx) = mpsc::channel(1);
registry.register(
"sandbox-1".to_string(),
"session-1".to_string(),
session_tx,
make_shutdown(),
);
let mut before_initialization = sandbox.clone();
registry.project_endpoint_status(&mut before_initialization);
let uninitialized_status = before_initialization.status.expect("status");
assert_eq!(uninitialized_status.endpoint_statuses, vec![unknown]);
assert_eq!(uninitialized_status.conditions, vec![ready.clone()]);
assert!(registry.initialize_endpoint_status_authority("sandbox-1", "session-1"));
registry.project_endpoint_status(&mut sandbox);
let initialized_status = sandbox.status.expect("status");
assert_eq!(initialized_status.endpoint_statuses, vec![endpoint]);
assert_eq!(initialized_status.conditions, vec![ready]);
}
fn sandbox_principal(sandbox_id: &str) -> Principal {
Principal::Sandbox(SandboxPrincipal {
sandbox_id: sandbox_id.to_string(),
crates/openshell-server/tests/common/mod.rs
+9
View File
@@ -53,6 +53,15 @@ pub struct TestOpenShell;
#[tonic::async_trait]
impl OpenShell for TestOpenShell {
async fn report_endpoint_status(
&self,
_request: tonic::Request<openshell_core::proto::ReportEndpointStatusRequest>,
) -> Result<Response<openshell_core::proto::ReportEndpointStatusResponse>, Status> {
Ok(Response::new(
openshell_core::proto::ReportEndpointStatusResponse {},
))
}
async fn begin_rootfs_tar_staging(
&self,
_request: tonic::Request<openshell_core::proto::BeginRootfsTarStagingRequest>,
crates/openshell-server/tests/supervisor_relay_integration.rs
+9
View File
@@ -48,6 +48,15 @@ struct RelayGateway {
#[tonic::async_trait]
impl OpenShell for RelayGateway {
async fn report_endpoint_status(
&self,
_request: tonic::Request<openshell_core::proto::ReportEndpointStatusRequest>,
) -> Result<Response<openshell_core::proto::ReportEndpointStatusResponse>, Status> {
Ok(Response::new(
openshell_core::proto::ReportEndpointStatusResponse {},
))
}
async fn begin_rootfs_tar_staging(
&self,
_request: tonic::Request<openshell_core::proto::BeginRootfsTarStagingRequest>,
crates/openshell-supervisor-network/Cargo.toml
+2 -2
View File
@@ -31,7 +31,7 @@ ipnet = "2"
miette = { workspace = true }
prost-types = { workspace = true }
rcgen = { workspace = true }
regorus = { version = "0.9", default-features = false, features = ["std", "arc", "glob"] }
regorus = { version = "0.9", default-features = false, features = ["std", "arc", "glob", "yaml"] }
reqwest = { workspace = true }
rustls = { workspace = true }
rustls-native-certs = { workspace = true }
@@ -42,6 +42,7 @@ serde_yml = { workspace = true }
sha1 = "0.10"
sha2 = { workspace = true }
spiffe = { workspace = true }
tempfile = "3"
thiserror = { workspace = true }
tokio = { workspace = true }
tokio-rustls = { workspace = true }
@@ -56,7 +57,6 @@ bundled-ca-roots = ["dep:webpki-roots"]
[dev-dependencies]
openshell-supervisor-middleware-builtins = { path = "../openshell-supervisor-middleware-builtins" }
tempfile = "3"
tonic = { workspace = true }
temp-env = "0.3"
tokio = { workspace = true, features = ["test-util"] }
crates/openshell-supervisor-network/src/host.rs
+358
View File
@@ -0,0 +1,358 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
//! Host-side proxy startup for compute drivers that do not run the Linux
//! in-sandbox supervisor.
//!
//! MXC uses this on Windows: MXC's `network.proxy` redirects sandbox egress to a
//! per-sandbox loopback listener in the gateway process, and this module starts
//! the existing OpenShell CONNECT proxy against the trimmed network-only
//! `SandboxPolicy`.
use std::net::SocketAddr;
use std::path::PathBuf;
use std::sync::Arc;
use miette::Result;
use openshell_core::activity::ActivitySender;
use openshell_core::denial::DenialEvent;
use openshell_core::policy::ProxyPolicy;
use openshell_core::proposals::AgentProposals;
use openshell_core::proto::SandboxPolicy as ProtoSandboxPolicy;
use openshell_core::provider_credentials::ProviderCredentialState;
use openshell_ocsf::{
ConfigStateChangeBuilder, SeverityId, StateId, StatusId, ctx::ctx as ocsf_ctx, ocsf_emit,
};
use tokio::sync::mpsc::UnboundedSender;
use crate::l7::tls::{
CertCache, ProxyTlsState, SandboxCa, build_upstream_client_config, read_system_ca_bundle,
write_ca_files,
};
use crate::opa::OpaEngine;
use crate::policy_local::PolicyLocalContext;
use crate::proxy::{ProxyHandle, ProxyIdentityMode};
/// Configuration for a host-side OpenShell CONNECT proxy.
pub struct HostProxyConfig {
/// Exact socket the compute driver will redirect sandbox egress to.
pub bind_addr: SocketAddr,
/// Network-only policy produced by the compute driver's policy split.
pub policy: ProtoSandboxPolicy,
/// Static process identity used when the platform cannot recover the
/// socket-owning sandbox process. Policy binaries must match this path for
/// L4/L7 allow rules to pass.
pub binary_path: PathBuf,
pub sandbox_id: Option<String>,
pub sandbox_name: Option<String>,
pub openshell_endpoint: Option<String>,
pub provider_credentials: Option<ProviderCredentialState>,
/// Shared feature state for the policy.local agent proposal surface.
pub agent_proposals: AgentProposals,
pub denial_tx: Option<UnboundedSender<DenialEvent>>,
pub activity_tx: Option<ActivitySender>,
}
/// RAII handle for a host-side proxy. Dropping it aborts the proxy accept loop.
pub struct HostProxyHandle {
proxy: ProxyHandle,
ca_file_paths: Option<(PathBuf, PathBuf)>,
#[allow(dead_code)]
tls_dir: Option<tempfile::TempDir>,
pub policy_local_ctx: Arc<PolicyLocalContext>,
}
impl HostProxyHandle {
#[must_use]
pub const fn http_addr(&self) -> Option<SocketAddr> {
self.proxy.http_addr()
}
#[must_use]
pub fn ca_file_paths(&self) -> Option<(PathBuf, PathBuf)> {
self.ca_file_paths.clone()
}
}
/// Start a host-side proxy for one sandbox.
///
/// Linux supervisor mode should continue to use `run::run_networking`; this API
/// is for host-side compute-driver integrations such as Windows MXC.
pub async fn start_host_proxy(config: HostProxyConfig) -> Result<HostProxyHandle> {
if !config.bind_addr.ip().is_loopback() {
return Err(miette::miette!(
"host proxy bind address must be loopback-only: {}",
config.bind_addr
));
}
if !config.policy.network_middlewares.is_empty() {
return Err(miette::miette!(
"host proxy cannot enforce network_middlewares without a middleware service registry; refusing to start with {} configured middleware(s)",
config.policy.network_middlewares.len()
));
}
let engine = Arc::new(OpaEngine::from_proto(&config.policy)?);
let (_workspace_tx, workspace_rx) = tokio::sync::watch::channel(String::new());
let policy_local_ctx = Arc::new(PolicyLocalContext::new(
Some(config.policy.clone()),
config.openshell_endpoint.clone(),
config
.sandbox_name
.clone()
.or_else(|| config.sandbox_id.clone()),
config.agent_proposals,
workspace_rx,
));
let (_ready_tx, ready_rx) = tokio::sync::watch::channel(true);
let proxy_policy = ProxyPolicy {
http_addr: Some(config.bind_addr),
};
let upstream_proxy_args = crate::upstream_proxy::UpstreamProxyArgs::default();
let (tls_state, ca_file_paths, tls_dir) = match tempfile::Builder::new()
.prefix("openshell-mxc-tls-")
.tempdir()
{
Ok(tls_dir) => match SandboxCa::generate() {
Ok(ca) => {
let system_ca_bundle = read_system_ca_bundle();
match write_ca_files(&ca, tls_dir.path(), &system_ca_bundle) {
Ok(paths) => match build_upstream_client_config(&system_ca_bundle) {
Ok(upstream_config) => {
let cert_cache = CertCache::new(ca);
let state = Arc::new(ProxyTlsState::new(cert_cache, upstream_config));
ocsf_emit!(
ConfigStateChangeBuilder::new(ocsf_ctx())
.severity(SeverityId::Informational)
.status(StatusId::Success)
.state(StateId::Enabled, "enabled")
.message(
"Host proxy TLS termination enabled: ephemeral CA generated"
)
.build()
);
(Some(state), Some(paths), Some(tls_dir))
}
Err(e) => {
ocsf_emit!(
ConfigStateChangeBuilder::new(ocsf_ctx())
.severity(SeverityId::High)
.status(StatusId::Failure)
.state(StateId::Disabled, "disabled")
.message(format!(
"Failed to build host proxy upstream TLS config, TLS termination disabled: {e}"
))
.build()
);
(None, None, Some(tls_dir))
}
},
Err(e) => {
ocsf_emit!(
ConfigStateChangeBuilder::new(ocsf_ctx())
.severity(SeverityId::High)
.status(StatusId::Failure)
.state(StateId::Disabled, "disabled")
.message(format!(
"Failed to write host proxy CA files, TLS termination disabled: {e}"
))
.build()
);
(None, None, Some(tls_dir))
}
}
}
Err(e) => {
ocsf_emit!(
ConfigStateChangeBuilder::new(ocsf_ctx())
.severity(SeverityId::High)
.status(StatusId::Failure)
.state(StateId::Disabled, "disabled")
.message(format!(
"Failed to generate host proxy ephemeral CA, TLS termination disabled: {e}"
))
.build()
);
(None, None, Some(tls_dir))
}
},
Err(e) => {
ocsf_emit!(
ConfigStateChangeBuilder::new(ocsf_ctx())
.severity(SeverityId::High)
.status(StatusId::Failure)
.state(StateId::Disabled, "disabled")
.message(format!(
"Failed to create host proxy TLS trust directory, TLS termination disabled: {e}"
))
.build()
);
(None, None, None)
}
};
let proxy = ProxyHandle::start_with_bind_addr(
&proxy_policy,
Some(config.bind_addr),
engine,
Arc::new(ProxyIdentityMode::static_binary(config.binary_path)?),
tls_state,
config.provider_credentials,
Some(policy_local_ctx.clone()),
config.denial_tx,
config.activity_tx,
ready_rx,
&upstream_proxy_args,
)
.await?;
Ok(HostProxyHandle {
proxy,
ca_file_paths,
tls_dir,
policy_local_ctx,
})
}
#[cfg(test)]
mod tests {
use std::time::Duration;
use openshell_core::proposals::AgentProposals;
use openshell_core::proto::{
MiddlewareEndpointSelector, NetworkMiddlewareConfig, SandboxPolicy as ProtoSandboxPolicy,
};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
use super::*;
fn test_config(bind_addr: SocketAddr, binary_path: PathBuf) -> HostProxyConfig {
HostProxyConfig {
bind_addr,
policy: ProtoSandboxPolicy {
version: 1,
..Default::default()
},
binary_path,
sandbox_id: Some("sandbox-123".to_string()),
sandbox_name: Some("agent-box".to_string()),
openshell_endpoint: None,
provider_credentials: None,
agent_proposals: AgentProposals::new(true),
denial_tx: None,
activity_tx: None,
}
}
#[tokio::test]
async fn rejects_non_loopback_bind_addr() {
let result = start_host_proxy(test_config(
([192, 0, 2, 1], 0).into(),
PathBuf::from("missing-agent.exe"),
))
.await;
let Err(err) = result else {
panic!("host proxy should reject non-loopback bind addresses");
};
assert!(
err.to_string().contains("loopback-only"),
"unexpected error: {err}"
);
}
#[tokio::test]
async fn rejects_middleware_policy_without_registry() {
let mut config = test_config(
([127, 0, 0, 1], 0).into(),
PathBuf::from("missing-agent.exe"),
);
config.policy.network_middlewares.insert(
"redactor".into(),
NetworkMiddlewareConfig {
name: "redactor".into(),
middleware: "openshell/regex".into(),
on_error: "fail_closed".into(),
endpoints: Some(MiddlewareEndpointSelector {
include: vec!["api.example.com".into()],
exclude: Vec::new(),
}),
..Default::default()
},
);
let Err(error) = start_host_proxy(config).await else {
panic!("host proxy must reject middleware without a registry");
};
assert!(
error
.to_string()
.contains("cannot enforce network_middlewares"),
"unexpected error: {error}"
);
}
#[tokio::test]
async fn starts_loopback_proxy_and_serves_policy_local() {
let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();
let binary = tempfile::NamedTempFile::new().unwrap();
std::fs::write(binary.path(), b"agent").unwrap();
let handle = start_host_proxy(test_config(
([127, 0, 0, 1], 0).into(),
binary.path().to_path_buf(),
))
.await
.unwrap();
let addr = handle.http_addr().expect("proxy should report bound addr");
assert!(addr.ip().is_loopback());
assert_ne!(addr.port(), 0);
let (ca_cert, bundle) = handle.ca_file_paths().expect("CA paths");
assert!(ca_cert.exists(), "standalone CA cert should exist");
assert!(bundle.exists(), "combined CA bundle should exist");
assert!(
std::fs::read_to_string(ca_cert)
.unwrap()
.contains("BEGIN CERTIFICATE")
);
assert!(
std::fs::read_to_string(bundle)
.unwrap()
.contains("BEGIN CERTIFICATE")
);
let mut client = TcpStream::connect(addr).await.unwrap();
client
.write_all(
b"GET http://policy.local/v1/policy/current HTTP/1.1\r\n\
Host: policy.local\r\n\
Connection: close\r\n\
\r\n",
)
.await
.unwrap();
let mut response = Vec::new();
tokio::time::timeout(Duration::from_secs(2), client.read_to_end(&mut response))
.await
.unwrap()
.unwrap();
let response = String::from_utf8(response).unwrap();
assert!(
response.starts_with("HTTP/1.1 200 OK"),
"unexpected response: {response}"
);
let (_, body) = response.split_once("\r\n\r\n").expect("response body");
let body: serde_json::Value = serde_json::from_str(body).unwrap();
assert_eq!(body["format"], "yaml");
assert!(
body["policy_yaml"]
.as_str()
.unwrap_or_default()
.contains("version: 1"),
"unexpected policy payload: {body}"
);
}
}
crates/openshell-supervisor-network/src/identity.rs
+29 -12
View File
@@ -21,10 +21,8 @@ use tracing::debug;
#[derive(Clone)]
struct FileFingerprint {
len: u64,
mtime_sec: i64,
mtime_nsec: i64,
ctime_sec: i64,
ctime_nsec: i64,
mtime: Option<(i64, i64)>,
ctime: Option<(i64, i64)>,
#[cfg(unix)]
dev: u64,
#[cfg(unix)]
@@ -33,12 +31,20 @@ struct FileFingerprint {
impl FileFingerprint {
fn from_metadata(metadata: &Metadata) -> Self {
#[cfg(unix)]
let (mtime, ctime) = (
Some((metadata.mtime(), metadata.mtime_nsec())),
Some((metadata.ctime(), metadata.ctime_nsec())),
);
#[cfg(not(unix))]
let (mtime, ctime) = (
metadata.modified().ok().and_then(system_time_parts),
metadata.created().ok().and_then(system_time_parts),
);
Self {
len: metadata.len(),
mtime_sec: metadata.mtime(),
mtime_nsec: metadata.mtime_nsec(),
ctime_sec: metadata.ctime(),
ctime_nsec: metadata.ctime_nsec(),
mtime,
ctime,
#[cfg(unix)]
dev: metadata.dev(),
#[cfg(unix)]
@@ -47,13 +53,24 @@ impl FileFingerprint {
}
}
#[cfg(not(unix))]
fn system_time_parts(time: std::time::SystemTime) -> Option<(i64, i64)> {
let duration = time.duration_since(std::time::UNIX_EPOCH).ok()?;
Some((
duration.as_secs() as i64,
i64::from(duration.subsec_nanos()),
))
}
impl PartialEq for FileFingerprint {
fn eq(&self, other: &Self) -> bool {
self.len == other.len
&& self.mtime_sec == other.mtime_sec
&& self.mtime_nsec == other.mtime_nsec
&& self.ctime_sec == other.ctime_sec
&& self.ctime_nsec == other.ctime_nsec
&& self.mtime.is_some()
&& other.mtime.is_some()
&& self.mtime == other.mtime
&& self.ctime.is_some()
&& other.ctime.is_some()
&& self.ctime == other.ctime
&& {
#[cfg(unix)]
{
crates/openshell-supervisor-network/src/l7/mod.rs
+383
View File
@@ -21,11 +21,18 @@ pub mod tls;
pub(crate) mod token_grant_injection;
pub(crate) mod websocket;
use openshell_core::endpoint_status::{
EndpointObservationContext, EndpointObservationHandle, EndpointObservationSender,
EndpointResult,
};
use openshell_core::mcp::McpProtocolVersion;
pub use openshell_policy::L7Protocol;
use openshell_policy::{
L7EndpointFields, validate_explicit_tcp_additional_fields, validate_l7_endpoint_semantics,
};
use std::sync::{Arc, Mutex};
use crate::opa::PolicyGenerationGuard;
pub(crate) fn build_credential_endpoint_mismatch_finding(
policy_name: &str,
@@ -108,6 +115,17 @@ pub enum EnforcementMode {
#[derive(Debug, Clone)]
pub struct L7EndpointConfig {
pub protocol: L7Protocol,
/// Opaque endpoint identity derived from the matched policy endpoint.
///
/// The value is added to the supervisor's internal OPA data and carried
/// through path selection unchanged. It is never derived from the concrete
/// request host because that host may have matched a wildcard selector.
pub endpoint_id: String,
/// Hash of the complete canonical policy that supplied this endpoint.
///
/// Internal OPA metadata binds a selected endpoint to its observation
/// inventory even while policy activation and inventory publication overlap.
pub policy_hash: String,
/// Optional endpoint-level HTTP path glob used to select between L7
/// protocols that share the same host:port.
pub path: String,
@@ -151,6 +169,102 @@ pub struct L7EndpointConfig {
pub signing_region: String,
}
/// Report the first network result for a selected tool endpoint.
///
/// Clones share the same pending handle, so only the first terminal result is
/// accepted. The lock is never held across an await, and reporting remains
/// nonblocking because the core sender uses a bounded `try_send` path.
#[derive(Clone)]
pub(crate) struct EndpointObserver {
sender: EndpointObservationSender,
observation: Arc<Mutex<Option<EndpointObservationHandle>>>,
uses_provider_credentials: bool,
policy_generation: Option<PolicyGenerationGuard>,
}
impl EndpointObserver {
/// Bind a selected endpoint to authority captured before request-state lookup.
///
/// Selection, provider material, and observation authority must describe the
/// same installation. A stale policy selection never acquires a new epoch.
pub(crate) fn begin_captured(
sender: Option<&EndpointObservationSender>,
config: &L7EndpointConfig,
context: Option<&EndpointObservationContext>,
provider_revision: Option<u64>,
guard: Option<&PolicyGenerationGuard>,
) -> Option<Self> {
if config.protocol != L7Protocol::Mcp
|| config.endpoint_id.is_empty()
|| config.policy_hash.is_empty()
{
// Only internal metadata from a complete MCP policy identifies
// the endpoint and configuration whose result can be published.
return None;
}
if guard.is_some_and(PolicyGenerationGuard::is_stale) {
// Equal policy hashes can recur after a reload; the pinned guard
// prevents an earlier selection from becoming current again.
return None;
}
let sender = sender?.clone();
let observation = sender.begin_captured(
context?,
config.endpoint_id.clone(),
&config.policy_hash,
provider_revision,
)?;
Some(Self {
sender,
observation: Arc::new(Mutex::new(Some(observation))),
uses_provider_credentials: config.provider_credentialed,
policy_generation: guard.cloned(),
})
}
/// Capture and bind an observation synchronously for isolated unit fixtures.
#[cfg(test)]
pub(crate) fn begin(
sender: Option<&EndpointObservationSender>,
config: &L7EndpointConfig,
) -> Option<Self> {
let context = sender.and_then(EndpointObservationSender::capture);
Self::begin_captured(sender, config, context.as_ref(), None, None)
}
/// Report the first terminal exchange result, dropping later attempts.
pub(crate) fn observe(&self, result: EndpointResult) {
let Ok(mut pending) = self.observation.lock() else {
// Status reporting cannot interrupt traffic if another observer panicked.
return;
};
let Some(observation) = pending.take() else {
// Another clone already reported the result for this exchange.
return;
};
if self
.policy_generation
.as_ref()
.is_some_and(PolicyGenerationGuard::is_stale)
{
// The request may finish after its policy was replaced but before
// the replacement inventory arrives. Its old result cannot fill
// that activation gap, including a return to equal policy values.
return;
}
let _accepted = self.sender.try_observe(observation, result);
}
/// Report an endpoint mismatch or an unavailable provider credential.
pub(crate) fn observe_credential_failure(&self, endpoint_mismatch: bool) {
if endpoint_mismatch {
self.observe(EndpointResult::PolicyDenied);
} else if self.uses_provider_credentials {
self.observe(EndpointResult::CredentialUnavailable);
}
}
}
/// Result of an L7 policy decision for a single request.
#[derive(Debug, Clone)]
pub struct L7Decision {
@@ -225,6 +339,8 @@ pub fn parse_l7_config(val: &regorus::Value) -> Option<L7EndpointConfig> {
let allow_uninspected_credentials =
get_object_bool(val, "allow_uninspected_credentials").unwrap_or(false);
let provider_credentialed = get_object_bool(val, "provider_credentialed").unwrap_or(false);
let endpoint_id = get_object_str(val, "endpoint_id").unwrap_or_default();
let policy_hash = get_object_str(val, "policy_hash").unwrap_or_default();
let websocket_graphql_policy =
protocol == L7Protocol::Websocket && endpoint_has_graphql_policy(val);
let graphql_max_body_bytes = get_object_u64(val, "graphql_max_body_bytes")
@@ -276,6 +392,8 @@ pub fn parse_l7_config(val: &regorus::Value) -> Option<L7EndpointConfig> {
Some(L7EndpointConfig {
protocol,
endpoint_id,
policy_hash,
path: get_object_str(val, "path").unwrap_or_default(),
tls,
enforcement,
@@ -1845,6 +1963,271 @@ mod tests {
assert_eq!(config.enforcement, EnforcementMode::Audit);
}
#[tokio::test]
async fn endpoint_observation_classifies_only_bound_credential_failure() {
use openshell_core::endpoint_status::{
EndpointConfigVersion, EndpointInventoryEntry, EndpointResult, EndpointStatusCommand,
endpoint_status_channel,
};
let endpoint_id = "endpoint:v1:credential".to_string();
let (sender, mut receiver) = endpoint_status_channel();
sender
.reset(
EndpointConfigVersion {
policy_hash: "policy".to_string(),
provider_env_revision: 1,
},
vec![EndpointInventoryEntry {
endpoint_id: endpoint_id.clone(),
uses_provider_credentials: true,
}],
)
.await
.expect("install MCP test inventory");
let _reset = receiver.recv().await.expect("inventory reset");
let value = regorus::Value::from_json_str(&format!(
r#"{{"protocol":"mcp","mcp_versions":["2025-11-25"],"endpoint_id":"{endpoint_id}","policy_hash":"policy","provider_credentialed":true}}"#
))
.expect("parse MCP config JSON");
let config = parse_l7_config(&value).expect("parse MCP config");
let observer = EndpointObserver::begin(Some(&sender), &config).expect("begin observation");
observer.observe_credential_failure(false);
assert!(matches!(
receiver.recv().await.expect("credential observation"),
EndpointStatusCommand::Observe {
result: EndpointResult::CredentialUnavailable,
..
}
));
}
#[tokio::test]
async fn endpoint_observation_rejects_old_policy_selection_after_inventory_reset() {
use openshell_core::endpoint_status::{
EndpointConfigVersion, EndpointInventoryEntry, endpoint_status_channel,
};
let endpoint_id = "endpoint:v1:shared";
let inventory = vec![EndpointInventoryEntry {
endpoint_id: endpoint_id.to_string(),
uses_provider_credentials: false,
}];
let (sender, mut receiver) = endpoint_status_channel();
let mut tracker = receiver.tracker();
sender
.reset(
EndpointConfigVersion {
policy_hash: "policy-a".to_string(),
provider_env_revision: 1,
},
inventory.clone(),
)
.await
.expect("install first policy inventory");
assert!(tracker.apply(receiver.recv().await.expect("first inventory reset")));
let value = regorus::Value::from_json_str(
r#"{"protocol":"mcp","mcp_versions":["2025-11-25"],"endpoint_id":"endpoint:v1:shared","policy_hash":"policy-a"}"#,
)
.expect("parse first policy selection");
let selected = parse_l7_config(&value).expect("select first policy endpoint");
// The replacement retains the endpoint address while changing policy.
// A delayed selection must not acquire the replacement's authority.
sender
.reset(
EndpointConfigVersion {
policy_hash: "policy-b".to_string(),
provider_env_revision: 1,
},
inventory,
)
.await
.expect("install replacement policy inventory");
assert!(tracker.apply(receiver.recv().await.expect("replacement inventory reset")));
if let Some(observer) = EndpointObserver::begin(Some(&sender), &selected) {
observer.observe(EndpointResult::PolicyDenied);
}
while let Ok(command) = receiver.try_recv() {
tracker.apply(command);
}
let snapshot = tracker
.snapshot()
.expect("replacement inventory remains installed");
assert_eq!(snapshot.config_version.policy_hash, "policy-b");
assert_eq!(snapshot.endpoints.len(), 1);
assert_eq!(
snapshot.endpoints[0].result,
EndpointResult::NoObservedExchange
);
}
#[tokio::test]
async fn endpoint_observation_rejects_new_policy_selection_before_inventory_reset() {
use openshell_core::endpoint_status::{
EndpointConfigVersion, EndpointInventoryEntry, endpoint_status_channel,
};
let (sender, mut receiver) = endpoint_status_channel();
let mut tracker = receiver.tracker();
sender
.reset(
EndpointConfigVersion {
policy_hash: "policy-a".to_string(),
provider_env_revision: 1,
},
vec![EndpointInventoryEntry {
endpoint_id: "endpoint:v1:shared".to_string(),
uses_provider_credentials: false,
}],
)
.await
.expect("install first policy inventory");
assert!(tracker.apply(receiver.recv().await.expect("first inventory reset")));
// The runtime can install the new policy before its inventory reset.
// The new policy's result must not be attributed to the old inventory.
let value = regorus::Value::from_json_str(
r#"{"protocol":"mcp","mcp_versions":["2025-11-25"],"endpoint_id":"endpoint:v1:shared","policy_hash":"policy-b"}"#,
)
.expect("parse replacement policy selection");
let selected = parse_l7_config(&value).expect("select replacement policy endpoint");
if let Some(observer) = EndpointObserver::begin(Some(&sender), &selected) {
observer.observe(EndpointResult::HttpResponseReceived);
}
while let Ok(command) = receiver.try_recv() {
tracker.apply(command);
}
let snapshot = tracker
.snapshot()
.expect("first inventory remains installed");
assert_eq!(snapshot.config_version.policy_hash, "policy-a");
assert_eq!(snapshot.endpoints.len(), 1);
assert_eq!(
snapshot.endpoints[0].result,
EndpointResult::NoObservedExchange
);
}
#[tokio::test]
async fn endpoint_observation_rejects_stale_policy_guards_without_disabling_fresh_requests() {
use openshell_core::endpoint_status::{
EndpointConfigVersion, EndpointInventoryEntry, endpoint_status_channel,
};
let policy = openshell_policy::restrictive_default_policy();
let engine = crate::opa::OpaEngine::from_proto(&policy).expect("test policy engine");
let (sender, mut receiver) = endpoint_status_channel();
let mut tracker = receiver.tracker();
sender
.reset(
EndpointConfigVersion {
policy_hash: "policy".to_string(),
provider_env_revision: 1,
},
vec![EndpointInventoryEntry {
endpoint_id: "endpoint:v1:shared".to_string(),
uses_provider_credentials: false,
}],
)
.await
.expect("install test inventory");
assert!(tracker.apply(receiver.recv().await.expect("inventory reset")));
let context = sender.capture().expect("capture installed authority");
let value = regorus::Value::from_json_str(
r#"{"protocol":"mcp","mcp_versions":["2025-11-25"],"endpoint_id":"endpoint:v1:shared","policy_hash":"policy"}"#,
)
.expect("parse selected MCP policy");
let selected = parse_l7_config(&value).expect("selected MCP endpoint");
let previous_guard = engine
.generation_guard(engine.current_generation())
.expect("capture initial policy guard");
let previous_observer = EndpointObserver::begin_captured(
Some(&sender),
&selected,
Some(&context),
Some(1),
Some(&previous_guard),
)
.expect("current policy can begin an observation");
// Runtime generations can advance while the canonical policy and
// inventory stay equal. Only new selections may use the new runtime.
engine
.reload_from_proto(&policy)
.expect("reinstall same policy");
assert!(
EndpointObserver::begin_captured(
Some(&sender),
&selected,
Some(&context),
Some(1),
Some(&previous_guard),
)
.is_none()
);
previous_observer.observe(EndpointResult::HttpResponseReceived);
while let Ok(command) = receiver.try_recv() {
tracker.apply(command);
}
assert_eq!(
tracker
.snapshot()
.expect("inventory remains installed")
.endpoints[0]
.result,
EndpointResult::NoObservedExchange,
);
let current_guard = engine
.generation_guard(engine.current_generation())
.expect("capture replacement policy guard");
let current_observer = EndpointObserver::begin_captured(
Some(&sender),
&selected,
Some(&context),
Some(1),
Some(&current_guard),
)
.expect("fresh selection stays observable without an inventory reset");
current_observer.observe(EndpointResult::HttpResponseReceived);
assert!(tracker.apply(receiver.recv().await.expect("fresh policy observation")));
assert_eq!(
tracker
.snapshot()
.expect("inventory remains installed")
.endpoints[0]
.result,
EndpointResult::HttpResponseReceived,
);
}
#[tokio::test]
async fn non_mcp_endpoint_never_begins_an_observation() {
use openshell_core::endpoint_status::{EndpointConfigVersion, endpoint_status_channel};
let (sender, _receiver) = endpoint_status_channel();
sender
.reset(
EndpointConfigVersion {
policy_hash: "policy".to_string(),
provider_env_revision: 1,
},
Vec::new(),
)
.await
.expect("install MCP test configuration");
let value = regorus::Value::from_json_str(
r#"{"protocol":"rest","endpoint_id":"endpoint:v1:not-mcp"}"#,
)
.expect("parse REST config JSON");
let config = parse_l7_config(&value).expect("parse REST config");
assert!(EndpointObserver::begin(Some(&sender), &config).is_none());
}
#[test]
fn parse_credential_signing_sigv4() {
let val = regorus::Value::from_json_str(
crates/openshell-supervisor-network/src/l7/relay.rs
+662 -18
View File
@@ -18,10 +18,11 @@ use crate::l7::middleware::{
};
use crate::l7::provider::{L7Provider, RelayOutcome};
use crate::l7::rest::WebSocketExtensionMode;
use crate::l7::{EnforcementMode, L7EndpointConfig, L7Protocol, L7RequestInfo};
use crate::l7::{EndpointObserver, EnforcementMode, L7EndpointConfig, L7Protocol, L7RequestInfo};
use crate::opa::{PolicyGenerationGuard, TunnelPolicyEngine};
use miette::{IntoDiagnostic, Result, miette};
use openshell_core::activity::{ActivitySender, try_record_activity};
use openshell_core::endpoint_status::{EndpointObservationSender, EndpointResult};
use openshell_core::secrets::{self, SecretResolver};
use openshell_ocsf::{
ActionId, ActivityId, DetectionFindingBuilder, DispositionId, Endpoint, FindingInfo,
@@ -80,6 +81,8 @@ pub struct L7EvalContext {
Option<Arc<dyn crate::l7::token_grant_injection::TokenGrantResolver>>,
/// Shared feature state for agent-driven policy proposals.
pub(crate) agent_proposals: openshell_core::proposals::AgentProposals,
/// Bounded, nonblocking sink for privacy-safe tool server endpoint outcomes.
pub(crate) endpoint_observation_tx: Option<EndpointObservationSender>,
}
fn request_default_port(ctx: &L7EvalContext) -> Option<u16> {
@@ -169,6 +172,7 @@ where
/// Enforce MCP request-version policy and emit a transport or policy rejection.
/// Non-MCP adapters share this entry point without changing their behavior.
/// Record endpoint version-policy denials before client response delivery.
pub(crate) async fn enforce_mcp_protocol_version<W>(
config: &L7EndpointConfig,
request: &crate::l7::provider::L7Request,
@@ -176,6 +180,7 @@ pub(crate) async fn enforce_mcp_protocol_version<W>(
client: &mut W,
ctx: &L7EvalContext,
redacted_target: &str,
observer: Option<&EndpointObserver>,
) -> Result<bool>
where
W: AsyncWrite + Unpin,
@@ -191,6 +196,12 @@ where
Ok(true)
}
Err(error) => {
if let Some(observer) = observer {
// Every explicit version-gate rejection is local to this MCP
// endpoint. Record it before delivery can fail; an invalid client
// version is not an upstream transport failure.
observer.observe(EndpointResult::PolicyDenied);
}
let reason = error.to_string();
let summary = l7_protocol_log_summary(None, Some(info));
ocsf_emit!(build_l7_request_event(
@@ -240,6 +251,7 @@ pub(crate) async fn enforce_final_mcp_protocol_version<W>(
client: &mut W,
ctx: &L7EvalContext,
redacted_target: &str,
observer: Option<&EndpointObserver>,
) -> Result<bool>
where
W: AsyncWrite + Unpin,
@@ -251,7 +263,16 @@ where
request,
crate::l7::jsonrpc::JsonRpcInspectionOptions::for_config(config),
)?;
enforce_mcp_protocol_version(config, request, &info, client, ctx, redacted_target).await
enforce_mcp_protocol_version(
config,
request,
&info,
client,
ctx,
redacted_target,
observer,
)
.await
}
fn build_request_authority_mismatch_event(ctx: &L7EvalContext) -> openshell_ocsf::OcsfEvent {
@@ -403,21 +424,56 @@ where
C: AsyncRead + AsyncWrite + Unpin,
U: AsyncRead + AsyncWrite + Unpin,
{
match crate::l7::rest::relay_http_request_with_options_guarded(
request, client, upstream, options,
relay_http_request_with_credential_rejection_observed(
request, client, upstream, options, ctx, None,
)
.await
}
async fn relay_http_request_with_credential_rejection_observed<C, U>(
request: &crate::l7::provider::L7Request,
client: &mut C,
upstream: &mut U,
options: crate::l7::rest::RelayRequestOptions<'_>,
ctx: &L7EvalContext,
observer: Option<&EndpointObserver>,
) -> Result<Option<RelayOutcome>>
where
C: AsyncRead + AsyncWrite + Unpin,
U: AsyncRead + AsyncWrite + Unpin,
{
match crate::l7::rest::relay_http_request_with_options_guarded_observed(
request, client, upstream, options, observer,
)
.await
{
Ok(outcome) => Ok(Some(outcome)),
Err(report) => {
if let Some(error) = report.downcast_ref::<secrets::body::BodyCredentialError>() {
// Body classification includes credentials for other destinations,
// so denial does not establish that this endpoint lacks credentials.
// Record the local decision before cleanup or client I/O can fail.
if let Some(observer) = observer {
observer.observe(EndpointResult::PolicyDenied);
}
let _ = upstream.shutdown().await;
reject_body_credential(client, *error).await?;
Ok(None)
} else if let Some(error) = report.downcast_ref::<secrets::UnresolvedPlaceholderError>()
{
if let Some(observer) = observer {
observer.observe_credential_failure(error.is_endpoint_mismatch());
}
reject_credential_resolution(client, ctx, error).await?;
Ok(None)
} else if report
.downcast_ref::<crate::l7::rest::CredentialUnavailableError>()
.is_some()
{
if let Some(observer) = observer {
observer.observe_credential_failure(false);
}
Err(report)
} else {
Err(report)
}
@@ -642,6 +698,12 @@ where
return Ok(());
}
// Pin observation authority before parsing can await or a later
// provider lookup can select state from another installation.
let observation_context = ctx
.endpoint_observation_tx
.as_ref()
.and_then(EndpointObservationSender::capture);
let mut req = match provider.parse_request(client).await {
Ok(Some(req)) => req,
Ok(None) => return Ok(()),
@@ -688,6 +750,13 @@ where
.await?;
return Ok(());
};
let observer = EndpointObserver::begin_captured(
ctx.endpoint_observation_tx.as_ref(),
config,
observation_context.as_ref(),
ctx.provider_credential_revision,
Some(engine.generation_guard()),
);
// The request was canonicalized before the matching config was known,
// so `allow_encoded_slash` was taken permissively across every config
// on this host:port. Re-check it against the config that actually
@@ -701,6 +770,9 @@ where
&& crate::l7::path::canonical_path_has_encoded_slash(&req.target)
{
let detail = "request-target contains an encoded '/' (%2F) which is not allowed on this endpoint";
if let Some(observer) = observer.as_ref() {
observer.observe(EndpointResult::PolicyDenied);
}
emit_parse_rejection(ctx, detail, engine_type_for_protocol(config.protocol));
crate::l7::rest::RestProvider::default()
.deny_with_redacted_target(
@@ -791,6 +863,9 @@ where
let redacted_target = match secrets::redact_target_for_policy(&req.target) {
Ok(target) => target,
Err(error) => {
if let Some(observer) = observer.as_ref() {
observer.observe_credential_failure(error.is_endpoint_mismatch());
}
reject_credential_resolution(client, ctx, &error).await?;
return Ok(());
}
@@ -804,8 +879,16 @@ where
jsonrpc: jsonrpc_info.clone(),
};
if let Some(info) = jsonrpc_info.as_ref()
&& !enforce_mcp_protocol_version(config, &req, info, client, ctx, &redacted_target)
.await?
&& !enforce_mcp_protocol_version(
config,
&req,
info,
client,
ctx,
&redacted_target,
observer.as_ref(),
)
.await?
{
return Ok(());
}
@@ -876,6 +959,9 @@ where
let req = match middleware_result? {
MiddlewareApplyResult::Allowed(request) => request,
MiddlewareApplyResult::Denied { denial, .. } => {
if let Some(observer) = observer.as_ref() {
observer.observe(EndpointResult::PolicyDenied);
}
let denied_request = crate::l7::provider::L7Request {
action: request_info.action.clone(),
target: redacted_target.clone(),
@@ -911,13 +997,30 @@ where
return Ok(());
}
};
if !enforce_final_mcp_protocol_version(config, &req, client, ctx, &redacted_target)
.await?
if !enforce_final_mcp_protocol_version(
config,
&req,
client,
ctx,
&redacted_target,
observer.as_ref(),
)
.await?
{
return Ok(());
}
let scoped_ctx = scoped_context_for_request(ctx, &req);
let ctx = scoped_ctx.as_ref().unwrap_or(ctx);
// Credential scoping can acquire a newer revision after middleware.
// Bind its actual snapshot to the original authority, never a fresh
// epoch; the earlier handle served only pre-forward local decisions.
let observer = EndpointObserver::begin_captured(
ctx.endpoint_observation_tx.as_ref(),
config,
observation_context.as_ref(),
ctx.provider_credential_revision,
Some(engine.generation_guard()),
);
let mut middleware_session = if let Some(chain) = websocket_chain.as_deref() {
let preflight = websocket_middleware_preflight(
&req,
@@ -951,7 +1054,7 @@ where
} else {
None
};
let outcome_result = relay_http_request_with_credential_rejection(
let outcome_result = relay_http_request_with_credential_rejection_observed(
&req,
client,
upstream,
@@ -975,6 +1078,7 @@ where
port: ctx.port,
},
ctx,
observer.as_ref(),
)
.await;
let outcome_result = match outcome_result {
@@ -1037,6 +1141,9 @@ where
}
}
} else {
if let Some(observer) = observer.as_ref() {
observer.observe(EndpointResult::PolicyDenied);
}
crate::l7::rest::RestProvider::default()
.deny_with_redacted_target(
&req,
@@ -1887,6 +1994,12 @@ where
return Ok(());
}
// Body inspection may await the caller while policy or provider state
// changes; a completed parse must retain its original authority.
let observation_context = ctx
.endpoint_observation_tx
.as_ref()
.and_then(EndpointObservationSender::capture);
let parsed = match crate::l7::jsonrpc::parse_jsonrpc_http_request(
client,
config.json_rpc_max_body_bytes,
@@ -1919,7 +2032,17 @@ where
let req = parsed.request;
let jsonrpc_info = parsed.info;
let observer = EndpointObserver::begin_captured(
ctx.endpoint_observation_tx.as_ref(),
config,
observation_context.as_ref(),
ctx.provider_credential_revision,
Some(engine.generation_guard()),
);
if !request_authority_matches_endpoint(&req, ctx) {
if let Some(observer) = observer.as_ref() {
observer.observe(EndpointResult::PolicyDenied);
}
reject_request_authority_mismatch(client, ctx).await?;
return Ok(());
}
@@ -1930,6 +2053,9 @@ where
let redacted_target = match secrets::redact_target_for_policy(&req.target) {
Ok(target) => target,
Err(error) => {
if let Some(observer) = observer.as_ref() {
observer.observe_credential_failure(error.is_endpoint_mismatch());
}
reject_credential_resolution(client, ctx, &error).await?;
return Ok(());
}
@@ -1942,8 +2068,16 @@ where
graphql: None,
jsonrpc: Some(jsonrpc_info.clone()),
};
if !enforce_mcp_protocol_version(config, &req, &jsonrpc_info, client, ctx, &redacted_target)
.await?
if !enforce_mcp_protocol_version(
config,
&req,
&jsonrpc_info,
client,
ctx,
&redacted_target,
observer.as_ref(),
)
.await?
{
return Ok(());
}
@@ -2023,6 +2157,9 @@ where
{
MiddlewareApplyResult::Allowed(request) => request,
MiddlewareApplyResult::Denied { denial, .. } => {
if let Some(observer) = observer.as_ref() {
observer.observe(EndpointResult::PolicyDenied);
}
let denied_request = crate::l7::provider::L7Request {
action: request_info.action.clone(),
target: redacted_target.clone(),
@@ -2058,19 +2195,36 @@ where
return Ok(());
}
};
if !enforce_final_mcp_protocol_version(config, &req, client, ctx, &redacted_target)
.await?
if !enforce_final_mcp_protocol_version(
config,
&req,
client,
ctx,
&redacted_target,
observer.as_ref(),
)
.await?
{
return Ok(());
}
let scoped_ctx = scoped_context_for_request(ctx, &req);
let ctx = scoped_ctx.as_ref().unwrap_or(ctx);
// The outgoing resolver and revision are one snapshot. Rebind using
// the original authority so a newly scoped provider revision cannot
// publish its result under the earlier inventory.
let observer = EndpointObserver::begin_captured(
ctx.endpoint_observation_tx.as_ref(),
config,
observation_context.as_ref(),
ctx.provider_credential_revision,
Some(engine.generation_guard()),
);
// Future MCP response/SSE introspection or rewrite would hook here
// before returning upstream bytes. The current policy schema has no
// trusted-annotations or version-profile field, so MCP responses and
// SSE streams are relayed unchanged; see McpOptions in
// proto/sandbox.proto for planned policy extensions.
let Some(outcome) = relay_http_request_with_credential_rejection(
let Some(outcome) = relay_http_request_with_credential_rejection_observed(
&req,
client,
upstream,
@@ -2082,6 +2236,7 @@ where
..Default::default()
},
ctx,
observer.as_ref(),
)
.await?
else {
@@ -2102,6 +2257,9 @@ where
}
}
} else {
if let Some(observer) = observer.as_ref() {
observer.observe(EndpointResult::PolicyDenied);
}
crate::l7::rest::RestProvider::default()
.deny_with_redacted_target(
&req,
@@ -3054,6 +3212,136 @@ mod tests {
assert!(!sent.contains("openshell:resolve:"));
}
async fn assert_body_denial_observation(
token: &str,
classifier: Option<&secrets::body::BodyCredentialClassifier>,
expected_reason: &str,
disconnect_client: bool,
) {
use openshell_core::endpoint_status::{
EndpointConfigVersion, EndpointInventoryEntry, EndpointStatusCommand,
endpoint_status_channel,
};
let (sender, mut receiver) = endpoint_status_channel();
sender
.reset(
EndpointConfigVersion {
policy_hash: "policy".into(),
provider_env_revision: 1,
},
vec![EndpointInventoryEntry {
endpoint_id: "endpoint:v1:body-test".into(),
uses_provider_credentials: true,
}],
)
.await
.expect("install endpoint inventory");
assert!(matches!(
receiver.recv().await,
Some(EndpointStatusCommand::Reset { .. })
));
let value = regorus::Value::from_json_str(
r#"{"protocol":"mcp","mcp_versions":["2025-11-25"],"endpoint_id":"endpoint:v1:body-test","policy_hash":"policy","provider_credentialed":true}"#,
)
.expect("parse endpoint configuration");
let config = crate::l7::parse_l7_config(&value).expect("parse MCP configuration");
let observer = EndpointObserver::begin(Some(&sender), &config).expect("begin observation");
let body = format!(r#"{{"input":"{token}"}}"#);
let req = crate::l7::provider::L7Request {
action: "POST".into(),
target: "/mcp".into(),
query_params: TestHashMap::new(),
raw_header: format!(
"POST /mcp HTTP/1.1\r\nHost: tools.example.test\r\nContent-Length: {}\r\n\r\n{body}",
body.len()
)
.into_bytes(),
body_length: crate::l7::provider::BodyLength::ContentLength(body.len() as u64),
};
let (mut client, caller) = tokio::io::duplex(4096);
let mut caller = Some(caller);
if disconnect_client {
// The denial must survive failure to deliver the local HTTP response.
drop(caller.take());
}
let (mut upstream, mut server) = tokio::io::duplex(4096);
let outcome = relay_http_request_with_credential_rejection_observed(
&req,
&mut client,
&mut upstream,
crate::l7::rest::RelayRequestOptions {
body_classifier: classifier,
deny_uninspected_credentials: true,
host: "tools.example.test",
port: 443,
..Default::default()
},
&L7EvalContext::default(),
Some(&observer),
)
.await;
if disconnect_client {
assert!(outcome.is_err());
} else {
assert!(outcome.expect("reject body locally").is_none());
}
drop(client);
if let Some(mut caller) = caller {
let mut response = String::new();
caller.read_to_string(&mut response).await.unwrap();
assert!(response.starts_with("HTTP/1.1 403 Forbidden"));
let body: serde_json::Value =
serde_json::from_str(response.split_once("\r\n\r\n").unwrap().1).unwrap();
assert_eq!(body["error"]["reason"], expected_reason);
assert!(!response.contains(token));
}
let mut sent = String::new();
server.read_to_string(&mut sent).await.unwrap();
assert!(!sent.contains("openshell:resolve:"));
assert!(!sent.contains("403 Forbidden"));
assert!(matches!(
receiver.try_recv().expect("body denial observation"),
EndpointStatusCommand::Observe {
result: EndpointResult::PolicyDenied,
..
}
));
assert!(receiver.try_recv().is_err(), "only one result per exchange");
}
#[tokio::test]
async fn body_denial_observation_reports_policy_denied_before_client_io() {
for disconnect_client in [false, true] {
assert_body_denial_observation(
"openshell:resolve:env:KEY",
None,
"classification_unavailable",
disconnect_client,
)
.await;
}
}
#[tokio::test]
async fn body_denial_observation_does_not_attribute_unrelated_unavailable_credential() {
let (state, _) = endpoint_mismatch_resolver(TestHashMap::from([(
"API_TOKEN".into(),
"private-test-secret".into(),
)]));
let (_, classifier, _) =
state.resolver_and_body_classifier_for_endpoint("tools.example.test", 443, "/mcp");
let classifier = classifier.expect("body classifier");
// This stale token belongs to allowed.example.test, not the observed server.
let token = "openshell:resolve:env:v999_API_TOKEN";
assert_eq!(
classifier.check(token),
Err(secrets::body::BodyCredentialError::KnownUnavailable)
);
assert_body_denial_observation(token, Some(&classifier), "known_unavailable", false).await;
}
const TEST_POLICY: &str = include_str!("../../data/sandbox-policy.rego");
fn endpoint_binding(identity: &str) -> StaticCredentialBinding {
@@ -7526,6 +7814,8 @@ network_policies:
fn encoded_slash_scoping_configs() -> Vec<L7EndpointConfig> {
let rest = |path: &str, allow_encoded_slash: bool| L7EndpointConfig {
protocol: L7Protocol::Rest,
endpoint_id: String::new(),
policy_hash: String::new(),
path: path.into(),
tls: crate::l7::TlsMode::Auto,
enforcement: EnforcementMode::Enforce,
@@ -7971,6 +8261,8 @@ network_policies:
.unwrap();
let configs = vec![L7EndpointConfig {
protocol: L7Protocol::Rest,
endpoint_id: String::new(),
policy_hash: String::new(),
path: "/ws".into(),
tls: crate::l7::TlsMode::Auto,
enforcement: EnforcementMode::Enforce,
@@ -8174,6 +8466,8 @@ network_policies:
.unwrap();
let configs = vec![L7EndpointConfig {
protocol: L7Protocol::Websocket,
endpoint_id: String::new(),
policy_hash: String::new(),
path: "/ws".into(),
tls: crate::l7::TlsMode::Auto,
enforcement: EnforcementMode::Enforce,
@@ -8301,6 +8595,8 @@ network_policies:
.unwrap();
let configs = vec![L7EndpointConfig {
protocol: L7Protocol::Websocket,
endpoint_id: String::new(),
policy_hash: String::new(),
path: "/graphql".into(),
tls: crate::l7::TlsMode::Auto,
enforcement: EnforcementMode::Enforce,
@@ -8917,6 +9213,348 @@ network_policies:
assert!(forwarded.is_empty(), "rejected request reached upstream");
}
const MCP_VERSION_DENIALS: &[(&str, &str, &str)] = &[
("", "403 Forbidden", "mcp_protocol_version_not_allowed"),
(
"MCP-Protocol-Version: \r\n",
"400 Bad Request",
"invalid_mcp_protocol_version_header",
),
(
"MCP-Protocol-Version: 2025-11-25\r\nMCP-Protocol-Version: 2025-11-25\r\n",
"400 Bad Request",
"invalid_mcp_protocol_version_header",
),
(
"MCP-Protocol-Version: 2026-07-28\r\n",
"400 Bad Request",
"unsupported_mcp_protocol_version",
),
(
"MCP-Protocol-Version: 2025-06-18\r\n",
"403 Forbidden",
"mcp_protocol_version_not_allowed",
),
];
#[tokio::test]
async fn endpoint_observation_binds_scoped_provider_revision() {
use openshell_core::endpoint_status::EndpointStatusCommand;
for route_selected in [false, true] {
for scoped_revision in [1, 2] {
let scenario = format!(
"route_selected={route_selected}, inventory_revision=1, parent_revision=1, scoped_revision={scoped_revision}"
);
let (mut config, engine, mut ctx) = mcp_test_relay_context();
config.provider_credentialed = true;
let mut receiver = install_mcp_test_observation(&mut config, &mut ctx).await;
// The tunnel retains revision 1 while request scoping may obtain
// revision 2 before its replacement inventory has been published.
ctx.provider_credential_revision = Some(1);
let credentials = ProviderCredentialState::from_bound_environment(
scoped_revision,
TestHashMap::from([("API_TOKEN".into(), "scoped-secret".into())]),
TestHashMap::new(),
TestHashMap::new(),
TestHashMap::from([(
"API_TOKEN".into(),
StaticCredentialBinding {
endpoints: vec![StaticCredentialEndpointBinding {
host: ctx.host.clone(),
port: u32::from(ctx.port),
path: "/mcp".into(),
}],
credential_identity: "provider:API_TOKEN".into(),
workload_credential_handle: String::new(),
},
)]),
Vec::new(),
)
.unwrap_or_else(|error| panic!("{scenario}: invalid provider fixture: {error}"));
// Endpoint-bound input must use the workload-issued handle;
// a canonical alias carries no authorized credential identity.
let placeholder = credentials.snapshot().child_env["API_TOKEN"].clone();
ctx.provider_credentials = Some(credentials);
let (mut app, mut client) = tokio::io::duplex(8192);
let (mut upstream, mut server) = tokio::io::duplex(8192);
let relay = tokio::spawn(async move {
if route_selected {
relay_with_route_selection(
&[config],
engine,
&mut client,
&mut upstream,
&ctx,
)
.await
} else {
relay_with_inspection(&config, engine, &mut client, &mut upstream, &ctx)
.await
}
});
let body = br#"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"test","version":"1"}}}"#;
let request = format!(
"POST /mcp HTTP/1.1\r\nHost: mcp.example.test:8000\r\nAuthorization: Bearer {placeholder}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n",
body.len()
);
app.write_all(request.as_bytes()).await.unwrap();
app.write_all(body).await.unwrap();
let mut forwarded = Vec::new();
tokio::time::timeout(std::time::Duration::from_secs(1), async {
while !forwarded.ends_with(body) {
let mut buffer = [0; 1024];
let count = server.read(&mut buffer).await.unwrap();
assert_ne!(
count, 0,
"{scenario}: request closed before forwarding its body"
);
forwarded.extend_from_slice(&buffer[..count]);
}
})
.await
.unwrap_or_else(|error| panic!("{scenario}: upstream exchange timed out: {error}"));
assert!(
String::from_utf8_lossy(&forwarded)
.contains("Authorization: Bearer scoped-secret\r\n"),
"{scenario}: endpoint-scoped credential was not injected"
);
server
.write_all(b"HTTP/1.1 204 No Content\r\nConnection: close\r\n\r\n")
.await
.unwrap();
tokio::time::timeout(std::time::Duration::from_secs(1), relay)
.await
.unwrap_or_else(|error| panic!("{scenario}: relay timed out: {error}"))
.unwrap_or_else(|error| panic!("{scenario}: relay task failed: {error}"))
.unwrap_or_else(|error| panic!("{scenario}: relay failed: {error}"));
if scoped_revision == 1 {
assert!(
matches!(
receiver.try_recv().unwrap_or_else(|error| panic!(
"{scenario}: matching revision observation missing: {error}"
)),
EndpointStatusCommand::Observe {
result: EndpointResult::HttpResponseReceived,
..
}
),
"{scenario}: matching revision produced an unexpected observation"
);
}
assert!(
receiver.try_recv().is_err(),
"{scenario}: another provider revision cannot supply this inventory's result"
);
}
}
}
#[tokio::test]
async fn endpoint_observation_records_mcp_version_policy_denial() {
use openshell_core::endpoint_status::EndpointStatusCommand;
for &(version_headers, status, response_code) in MCP_VERSION_DENIALS {
for route_selected in [false, true] {
for disconnect_client in [false, true] {
let (mut config, engine, mut ctx) = mcp_test_relay_context();
let mut receiver = install_mcp_test_observation(&mut config, &mut ctx).await;
let (mut app, mut client) = tokio::io::duplex(8192);
let (mut upstream, mut server) = tokio::io::duplex(8192);
let body = br#"{"jsonrpc":"2.0","id":7,"result":{"ok":true}}"#;
let request = format!(
"POST /mcp HTTP/1.1\r\nHost: mcp.example.test:8000\r\nContent-Type: application/json\r\n{version_headers}Content-Length: {}\r\nConnection: close\r\n\r\n",
body.len()
);
app.write_all(request.as_bytes()).await.unwrap();
app.write_all(body).await.unwrap();
let mut app = Some(app);
if disconnect_client {
// The buffered request remains readable after its caller
// disconnects, but delivering the local denial must fail.
drop(app.take());
}
let result = tokio::time::timeout(std::time::Duration::from_secs(1), async {
if route_selected {
relay_with_route_selection(
&[config],
engine,
&mut client,
&mut upstream,
&ctx,
)
.await
} else {
relay_with_inspection(&config, engine, &mut client, &mut upstream, &ctx)
.await
}
})
.await
.expect("version policy must reject before upstream I/O");
assert_eq!(result.is_err(), disconnect_client);
drop(client);
drop(upstream);
if let Some(mut app) = app {
let mut response = String::new();
app.read_to_string(&mut response).await.unwrap();
assert!(
response.starts_with(&format!("HTTP/1.1 {status}")),
"{response}"
);
assert!(response.contains(response_code), "{response}");
}
let mut sent = Vec::new();
server.read_to_end(&mut sent).await.unwrap();
assert!(sent.is_empty(), "disallowed version reached upstream");
assert!(matches!(
receiver
.try_recv()
.expect("version policy denial observation"),
EndpointStatusCommand::Observe {
result: EndpointResult::PolicyDenied,
..
}
));
assert!(receiver.try_recv().is_err(), "one result per exchange");
}
}
}
}
async fn install_mcp_test_observation(
config: &mut L7EndpointConfig,
ctx: &mut L7EvalContext,
) -> openshell_core::endpoint_status::EndpointStatusReceiver {
use openshell_core::endpoint_status::{
EndpointConfigVersion, EndpointInventoryEntry, EndpointStatusCommand,
endpoint_status_channel,
};
config.endpoint_id = "endpoint:v1:mcp-version".into();
config.policy_hash = "mcp-version-policy".into();
config.mcp_versions = vec![openshell_core::mcp::McpProtocolVersion::V2025_11_25];
let (sender, mut receiver) = endpoint_status_channel();
sender
.reset(
EndpointConfigVersion {
policy_hash: config.policy_hash.clone(),
provider_env_revision: 1,
},
vec![EndpointInventoryEntry {
endpoint_id: config.endpoint_id.clone(),
uses_provider_credentials: config.provider_credentialed,
}],
)
.await
.expect("install MCP endpoint inventory");
assert!(matches!(
receiver.recv().await,
Some(EndpointStatusCommand::Reset { .. })
));
ctx.endpoint_observation_tx = Some(sender);
receiver
}
#[tokio::test]
async fn endpoint_observation_records_final_mcp_version_policy_denial() {
use openshell_core::endpoint_status::EndpointStatusCommand;
for &(version_headers, status, response_code) in MCP_VERSION_DENIALS {
for disconnect_client in [false, true] {
let (mut config, _, mut ctx) = mcp_test_relay_context();
let mut receiver = install_mcp_test_observation(&mut config, &mut ctx).await;
// One request handle survives initialize's version exemption and
// the final check after a middleware changes the outgoing body.
let observer =
EndpointObserver::begin(ctx.endpoint_observation_tx.as_ref(), &config)
.expect("begin transformed request observation");
let buffered_request = |body: &[u8]| {
let mut raw_header = format!(
"POST /mcp HTTP/1.1\r\nHost: mcp.example.test:8000\r\nContent-Type: application/json\r\n{version_headers}Content-Length: {}\r\nConnection: close\r\n\r\n",
body.len()
)
.into_bytes();
raw_header.extend_from_slice(body);
crate::l7::provider::L7Request {
action: "POST".into(),
target: "/mcp".into(),
query_params: TestHashMap::new(),
raw_header,
body_length: crate::l7::provider::BodyLength::ContentLength(
body.len() as u64
),
}
};
let initialize = buffered_request(
br#"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"test","version":"1"}}}"#,
);
let (mut client, app) = tokio::io::duplex(2048);
assert!(
enforce_final_mcp_protocol_version(
&config,
&initialize,
&mut client,
&ctx,
"/mcp",
Some(&observer),
)
.await
.expect("initialize version exemption")
);
assert!(
receiver.try_recv().is_err(),
"allowed request is not a terminal result"
);
let rewritten =
buffered_request(br#"{"jsonrpc":"2.0","id":2,"method":"tools/list"}"#);
let mut app = Some(app);
if disconnect_client {
// Dropping the peer makes the final rejection undeliverable.
drop(app.take());
}
let result = enforce_final_mcp_protocol_version(
&config,
&rewritten,
&mut client,
&ctx,
"/mcp",
Some(&observer),
)
.await;
if disconnect_client {
assert!(
result.is_err(),
"local denial cannot reach a disconnected client"
);
} else {
assert!(!result.expect("reject rewritten request version"));
}
drop(client);
if let Some(mut app) = app {
let mut response = String::new();
app.read_to_string(&mut response).await.unwrap();
assert!(
response.starts_with(&format!("HTTP/1.1 {status}")),
"{response}"
);
assert!(response.contains(response_code), "{response}");
}
assert!(matches!(
receiver
.try_recv()
.expect("final version policy denial observation"),
EndpointStatusCommand::Observe {
result: EndpointResult::PolicyDenied,
..
}
));
assert!(receiver.try_recv().is_err(), "one result per exchange");
}
}
}
#[tokio::test]
async fn final_mcp_version_check_reclassifies_a_rewritten_initialize_body() {
let (config, _, ctx) = mcp_test_relay_context();
@@ -8936,10 +9574,16 @@ network_policies:
};
let (mut client, mut relay_client) = tokio::io::duplex(2048);
let allowed =
enforce_final_mcp_protocol_version(&config, &request, &mut relay_client, &ctx, "/mcp")
.await
.expect("final request inspection");
let allowed = enforce_final_mcp_protocol_version(
&config,
&request,
&mut relay_client,
&ctx,
"/mcp",
None,
)
.await
.expect("final request inspection");
assert!(
!allowed,
"rewritten non-initialize request must require a version"
crates/openshell-supervisor-network/src/l7/rest.rs
+508 -22
View File
@@ -7,11 +7,13 @@
//! policy, and relays allowed requests to upstream. Handles Content-Length
//! and chunked transfer encoding for body framing.
use crate::l7::EndpointObserver;
use crate::l7::provider::{BodyLength, L7Provider, L7Request, RelayOutcome};
use crate::opa::PolicyGenerationGuard;
use aws_sigv4::http_request::SignableBody;
use base64::Engine as _;
use miette::{IntoDiagnostic, Result, miette};
use openshell_core::endpoint_status::EndpointResult;
use openshell_core::proto::{ExistingHeaderAction, HeaderMutation, header_mutation};
use openshell_core::secrets::{
SecretResolver, contains_reserved_credential_marker, contains_reserved_credential_marker_bytes,
@@ -768,6 +770,31 @@ pub(crate) struct CredentialGenerationGuard<'a> {
revision: u64,
}
/// Typed marker for provider credential material that cannot be used.
///
/// The type deliberately carries no secret or raw provider error, allowing
/// callers to classify the outcome without inspecting error strings.
#[derive(Debug)]
pub(crate) struct CredentialUnavailableError {
reason: &'static str,
}
impl CredentialUnavailableError {
fn new(reason: &'static str) -> Self {
Self { reason }
}
}
impl fmt::Display for CredentialUnavailableError {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter.write_str(self.reason)
}
}
impl std::error::Error for CredentialUnavailableError {}
impl miette::Diagnostic for CredentialUnavailableError {}
impl<'a> CredentialGenerationGuard<'a> {
pub(crate) fn new(
state: &'a openshell_core::provider_credentials::ProviderCredentialState,
@@ -780,9 +807,9 @@ impl<'a> CredentialGenerationGuard<'a> {
if self.state.revision() == self.revision {
Ok(())
} else {
Err(miette!(
"provider credential generation changed before upstream write"
))
Err(miette::Report::new(CredentialUnavailableError::new(
"provider credential generation changed before upstream write",
)))
}
}
}
@@ -800,6 +827,37 @@ pub(crate) async fn relay_http_request_with_options_guarded<C, U>(
upstream: &mut U,
options: RelayRequestOptions<'_>,
) -> Result<RelayOutcome>
where
C: AsyncRead + AsyncWrite + Unpin,
U: AsyncRead + AsyncWrite + Unpin,
{
relay_http_request_with_options_guarded_observed(req, client, upstream, options, None).await
}
/// Relay one request while attributing only terminal, privacy-safe MCP outcomes.
pub(crate) async fn relay_http_request_with_options_guarded_observed<C, U>(
req: &L7Request,
client: &mut C,
upstream: &mut U,
options: RelayRequestOptions<'_>,
observer: Option<&EndpointObserver>,
) -> Result<RelayOutcome>
where
C: AsyncRead + AsyncWrite + Unpin,
U: AsyncRead + AsyncWrite + Unpin,
{
let mut upstream = ObservedUpstream { upstream, observer };
relay_http_request_with_options_guarded_inner(req, client, &mut upstream, options, observer)
.await
}
async fn relay_http_request_with_options_guarded_inner<C, U>(
req: &L7Request,
client: &mut C,
upstream: &mut U,
options: RelayRequestOptions<'_>,
observer: Option<&EndpointObserver>,
) -> Result<RelayOutcome>
where
C: AsyncRead + AsyncWrite + Unpin,
U: AsyncRead + AsyncWrite + Unpin,
@@ -1066,15 +1124,15 @@ where
openshell_ocsf::ocsf_emit!(event);
}
_ => {
return Err(miette!(
"SigV4 signing configured but AWS credentials not found in provider"
));
return Err(miette::Report::new(CredentialUnavailableError::new(
"SigV4 signing configured but AWS credentials not found in provider",
)));
}
}
} else {
return Err(miette!(
"SigV4 signing configured but no secret resolver available"
));
return Err(miette::Report::new(CredentialUnavailableError::new(
"SigV4 signing configured but no secret resolver available",
)));
}
} else if options.request_body_credential_rewrite {
let body = collect_and_rewrite_request_body(
@@ -1153,6 +1211,7 @@ where
websocket_extensions: options.websocket_extensions,
websocket: websocket_response,
client_requested_upgrade,
observer,
},
)
.await?;
@@ -1172,6 +1231,71 @@ fn ensure_body_generation_current(options: RelayRequestOptions<'_>) -> Result<()
Ok(())
}
/// Upstream-only I/O adapter that cannot accidentally classify client errors.
///
/// Once a valid response status consumes the observation token, later stream
/// failures are ignored and cannot overwrite the endpoint's first network result.
struct ObservedUpstream<'a, U> {
upstream: &'a mut U,
observer: Option<&'a EndpointObserver>,
}
impl<U: AsyncRead + Unpin> AsyncRead for ObservedUpstream<'_, U> {
fn poll_read(
self: std::pin::Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
buf: &mut tokio::io::ReadBuf<'_>,
) -> std::task::Poll<std::io::Result<()>> {
let this = self.get_mut();
let result = std::pin::Pin::new(&mut *this.upstream).poll_read(cx, buf);
if matches!(result, std::task::Poll::Ready(Err(_)))
&& let Some(observer) = this.observer
{
observer.observe(EndpointResult::TransportFailed);
}
result
}
}
impl<U: AsyncWrite + Unpin> AsyncWrite for ObservedUpstream<'_, U> {
fn poll_write(
self: std::pin::Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
buf: &[u8],
) -> std::task::Poll<std::io::Result<usize>> {
let this = self.get_mut();
let result = std::pin::Pin::new(&mut *this.upstream).poll_write(cx, buf);
if matches!(result, std::task::Poll::Ready(Err(_)))
&& let Some(observer) = this.observer
{
observer.observe(EndpointResult::TransportFailed);
}
result
}
fn poll_flush(
self: std::pin::Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
) -> std::task::Poll<std::io::Result<()>> {
let this = self.get_mut();
let result = std::pin::Pin::new(&mut *this.upstream).poll_flush(cx);
if matches!(result, std::task::Poll::Ready(Err(_)))
&& let Some(observer) = this.observer
{
observer.observe(EndpointResult::TransportFailed);
}
result
}
fn poll_shutdown(
self: std::pin::Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
) -> std::task::Poll<std::io::Result<()>> {
let this = self.get_mut();
std::pin::Pin::new(&mut *this.upstream).poll_shutdown(cx)
}
}
async fn write_body_bytes<W: AsyncWrite + Unpin>(
writer: &mut W,
bytes: &[u8],
@@ -3110,18 +3234,20 @@ fn find_crlf(buf: &[u8], start: usize) -> Option<usize> {
}
#[derive(Clone)]
struct RelayResponseOptions {
struct RelayResponseOptions<'a> {
websocket_extensions: WebSocketExtensionMode,
client_requested_upgrade: bool,
websocket: Option<WebSocketResponseValidation>,
observer: Option<&'a EndpointObserver>,
}
impl Default for RelayResponseOptions {
impl Default for RelayResponseOptions<'_> {
fn default() -> Self {
Self {
websocket_extensions: WebSocketExtensionMode::Preserve,
client_requested_upgrade: true,
websocket: None,
observer: None,
}
}
}
@@ -3130,7 +3256,7 @@ async fn relay_response<U, C>(
request_method: &str,
upstream: &mut U,
client: &mut C,
options: RelayResponseOptions,
options: RelayResponseOptions<'_>,
) -> Result<RelayOutcome>
where
U: AsyncRead + Unpin,
@@ -3140,14 +3266,49 @@ where
let mut buf = Vec::with_capacity(4096);
let mut tmp = [0u8; 1024];
// Read response headers
loop {
if buf.len() > MAX_HEADER_BYTES {
let mut informational_header_bytes = 0;
let header_end = loop {
let header_end = buf
.windows(4)
.position(|w| w == b"\r\n\r\n")
.map(|end| end + 4);
let remaining_header_bytes = MAX_HEADER_BYTES - informational_header_bytes;
if header_end.is_some_and(|end| end > remaining_header_bytes)
|| (header_end.is_none() && buf.len() >= remaining_header_bytes)
{
if let Some(observer) = options.observer.as_ref() {
observer.observe(EndpointResult::TransportFailed);
}
return Err(miette!("HTTP response headers exceed limit"));
}
if let Some(header_end) = header_end {
let header_str = String::from_utf8_lossy(&buf[..header_end]);
if matches!(
parse_observed_http_status_code(&header_str),
Some(100 | 102..=199)
) {
// Informational responses keep this exchange and its observation
// pending. Preserve any coalesced final response, and charge every
// header block to one budget so repeated 1xx frames stay bounded.
informational_header_bytes += header_end;
client
.write_all(&buf[..header_end])
.await
.into_diagnostic()?;
client.flush().await.into_diagnostic()?;
buf.drain(..header_end);
continue;
}
// A 101 response transfers ownership to the upgrade handler below.
break header_end;
}
let n = upstream.read(&mut tmp).await.into_diagnostic()?;
if n == 0 {
if let Some(observer) = options.observer.as_ref() {
observer.observe(EndpointResult::TransportFailed);
}
// Upstream closed — forward whatever we have
if !buf.is_empty() {
client.write_all(&buf).await.into_diagnostic()?;
@@ -3155,16 +3316,25 @@ where
return Ok(RelayOutcome::Consumed);
}
buf.extend_from_slice(&tmp[..n]);
if buf.windows(4).any(|w| w == b"\r\n\r\n") {
break;
}
}
let header_end = buf.windows(4).position(|w| w == b"\r\n\r\n").unwrap() + 4;
};
// Parse response framing
let header_str = String::from_utf8_lossy(&buf[..header_end]);
let observed_status = parse_observed_http_status_code(&header_str);
if let Some(observer) = options.observer.as_ref() {
match observed_status {
Some(status_code) if status_code < 400 => {
observer.observe(EndpointResult::HttpResponseReceived);
}
Some(_) => observer.observe(EndpointResult::UpstreamRejected),
None => {
observer.observe(EndpointResult::TransportFailed);
}
}
}
// Preserve the relay's legacy framing fallback for non-MCP traffic and
// malformed upstream responses. Endpoint status uses the strict parser above,
// so this fallback can never become a false successful observation.
let status_code = parse_status_code(&header_str).unwrap_or(200);
let server_wants_close = parse_connection_close(&header_str);
let event_stream = response_is_event_stream(&header_str);
@@ -3308,6 +3478,23 @@ fn parse_status_code(headers: &str) -> Option<u16> {
code_str.parse().ok()
}
/// Parse a syntactically valid HTTP/1.x response status for endpoint reporting.
fn parse_observed_http_status_code(headers: &str) -> Option<u16> {
let status_line = headers.lines().next()?;
let mut fields = status_line.split_whitespace();
match fields.next()? {
"HTTP/1.0" | "HTTP/1.1" => {}
_ => return None,
}
let code = fields.next()?;
if code.len() != 3 || !code.bytes().all(|byte| byte.is_ascii_digit()) {
return None;
}
code.parse()
.ok()
.filter(|status| (100..=999).contains(status))
}
/// Check if the response headers contain `Connection: close`.
fn parse_connection_close(headers: &str) -> bool {
for line in headers.lines().skip(1) {
@@ -3629,6 +3816,7 @@ mod tests {
use super::*;
use crate::opa::OpaEngine;
use flate2::{Compress, Compression, Decompress, FlushCompress, FlushDecompress, Status};
use openshell_core::endpoint_status::{EndpointStatusCommand, EndpointStatusReceiver};
use openshell_core::proposals::AgentProposals;
use openshell_core::secrets::SecretResolver;
use std::pin::Pin;
@@ -5473,6 +5661,304 @@ mod tests {
assert_eq!(parse_status_code(""), None);
}
async fn test_endpoint_observer() -> (EndpointObserver, EndpointStatusReceiver) {
use openshell_core::endpoint_status::{
EndpointConfigVersion, EndpointInventoryEntry, endpoint_status_channel,
};
let endpoint_id = "endpoint:v1:test".to_string();
let (sender, mut receiver) = endpoint_status_channel();
sender
.reset(
EndpointConfigVersion {
policy_hash: "policy".to_string(),
provider_env_revision: 1,
},
vec![EndpointInventoryEntry {
endpoint_id: endpoint_id.clone(),
uses_provider_credentials: true,
}],
)
.await
.expect("install MCP test inventory");
let reset = receiver.recv().await.expect("inventory reset");
assert!(matches!(reset, EndpointStatusCommand::Reset { .. }));
let value = regorus::Value::from_json_str(&format!(
r#"{{"protocol":"mcp","mcp_versions":["2025-11-25"],"endpoint_id":"{endpoint_id}","policy_hash":"policy","provider_credentialed":true}}"#
))
.expect("parse MCP config JSON");
let config = crate::l7::parse_l7_config(&value).expect("parse MCP observation config");
let observer =
EndpointObserver::begin(Some(&sender), &config).expect("begin MCP observation");
(observer, receiver)
}
async fn assert_observed_response_result(response: &'static [u8], expected: EndpointResult) {
let (observer, mut receiver) = test_endpoint_observer().await;
let (mut upstream, mut upstream_peer) = tokio::io::duplex(4096);
let writer = tokio::spawn(async move {
upstream_peer
.write_all(response)
.await
.expect("write response");
});
let mut client = tokio::io::sink();
relay_response(
"POST",
&mut upstream,
&mut client,
RelayResponseOptions {
observer: Some(&observer),
..Default::default()
},
)
.await
.expect("relay observed response");
writer.await.expect("join upstream writer");
let command = receiver.recv().await.expect("endpoint observation");
assert!(matches!(
command,
EndpointStatusCommand::Observe { result, .. } if result == expected
));
}
#[tokio::test]
async fn endpoint_observation_records_http_response_at_header_receipt() {
assert_observed_response_result(
b"HTTP/1.1 204 No Content\r\n\r\n",
EndpointResult::HttpResponseReceived,
)
.await;
}
#[tokio::test]
async fn endpoint_observation_classifies_upstream_http_rejections() {
for response in [
&b"HTTP/1.1 401 Unauthorized\r\nContent-Length: 0\r\n\r\n"[..],
&b"HTTP/1.1 503 Service Unavailable\r\nContent-Length: 0\r\n\r\n"[..],
] {
assert_observed_response_result(response, EndpointResult::UpstreamRejected).await;
}
}
#[tokio::test]
async fn endpoint_observation_waits_for_final_response_after_coalesced_interim() {
let response = b"HTTP/1.1 103 Early Hints\r\n\r\nHTTP/1.1 100 Continue\r\n\r\nHTTP/1.1 503 Service Unavailable\r\nContent-Length: 4\r\n\r\ndeny";
let (observer, mut receiver) = test_endpoint_observer().await;
let mut upstream = &response[..];
let mut client = Vec::new();
let outcome = relay_response(
"POST",
&mut upstream,
&mut client,
RelayResponseOptions {
observer: Some(&observer),
..Default::default()
},
)
.await
.expect("relay informational and final responses");
assert!(matches!(
receiver.try_recv().expect("final response observation"),
EndpointStatusCommand::Observe {
result: EndpointResult::UpstreamRejected,
..
}
));
assert_eq!(
client, response,
"preserve coalesced final headers and body"
);
assert!(matches!(outcome, RelayOutcome::Reusable));
assert!(receiver.try_recv().is_err(), "one result per exchange");
}
#[tokio::test]
async fn endpoint_observation_relays_split_interim_and_final_headers() {
let first = b"HTTP/1.1 100 Continue\r\n\r\nHTTP/1.1 ";
let last = b"200 OK\r\nContent-Length: 4\r\n\r\nbody";
let (observer, mut receiver) = test_endpoint_observer().await;
// Separate reads split the final status line after the complete interim
// block; draining interim bytes must preserve the partial final header.
let mut upstream = first.as_slice().chain(last.as_slice());
let mut client = Vec::new();
relay_response(
"POST",
&mut upstream,
&mut client,
RelayResponseOptions {
observer: Some(&observer),
..Default::default()
},
)
.await
.expect("relay split informational and final responses");
assert_eq!(client, [first.as_slice(), last.as_slice()].concat());
assert!(matches!(
receiver.try_recv().expect("final response observation"),
EndpointStatusCommand::Observe {
result: EndpointResult::HttpResponseReceived,
..
}
));
assert!(receiver.try_recv().is_err(), "one result per exchange");
}
#[tokio::test]
async fn endpoint_observation_reports_transport_failure_after_interim_eof() {
for response in [
&b"HTTP/1.1 100 Continue\r\n\r\n"[..],
&b"HTTP/1.1 103 Early Hints\r\n\r\n"[..],
] {
assert_observed_response_result(response, EndpointResult::TransportFailed).await;
}
}
#[tokio::test]
async fn endpoint_observation_bounds_informational_response_headers() {
let interim = b"HTTP/1.1 103 Early Hints\r\n\r\n";
let mut response = interim.repeat(MAX_HEADER_BYTES / interim.len() + 1);
response.extend_from_slice(b"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r\n");
let (observer, mut receiver) = test_endpoint_observer().await;
let mut upstream = response.as_slice();
let mut client = Vec::new();
relay_response(
"POST",
&mut upstream,
&mut client,
RelayResponseOptions {
observer: Some(&observer),
..Default::default()
},
)
.await
.expect_err("informational headers must share a bounded exchange budget");
assert!(matches!(
receiver.try_recv().expect("header limit observation"),
EndpointStatusCommand::Observe {
result: EndpointResult::TransportFailed,
..
}
));
assert!(client.len() <= MAX_HEADER_BYTES);
assert!(receiver.try_recv().is_err(), "one result per exchange");
}
#[tokio::test]
async fn endpoint_observation_preserves_response_result_after_body_read_error() {
struct FailedReader;
impl AsyncRead for FailedReader {
fn poll_read(
self: Pin<&mut Self>,
_cx: &mut Context<'_>,
_buf: &mut ReadBuf<'_>,
) -> Poll<std::io::Result<()>> {
Poll::Ready(Err(std::io::Error::new(
std::io::ErrorKind::ConnectionReset,
"upstream response body interrupted",
)))
}
}
for (headers, expected) in [
(
&b"HTTP/1.1 200 OK\r\nContent-Length: 4\r\n\r\n"[..],
EndpointResult::HttpResponseReceived,
),
(
&b"HTTP/1.1 503 Service Unavailable\r\nContent-Length: 4\r\n\r\n"[..],
EndpointResult::UpstreamRejected,
),
] {
let (observer, mut receiver) = test_endpoint_observer().await;
let mut response = headers.chain(FailedReader);
let mut upstream = ObservedUpstream {
upstream: &mut response,
observer: Some(&observer),
};
let mut client = Vec::new();
relay_response(
"POST",
&mut upstream,
&mut client,
RelayResponseOptions {
observer: Some(&observer),
..Default::default()
},
)
.await
.expect_err("response body read must fail");
assert_eq!(client, headers);
assert!(matches!(
receiver.try_recv().expect("HTTP response observation"),
EndpointStatusCommand::Observe { result, .. } if result == expected
));
assert!(
receiver.try_recv().is_err(),
"body error must not replace the response result"
);
}
}
#[tokio::test]
async fn endpoint_observation_rejects_malformed_http_status() {
assert_observed_response_result(
b"not-http 200 nope\r\nContent-Length: 0\r\n\r\n",
EndpointResult::TransportFailed,
)
.await;
}
#[tokio::test]
async fn endpoint_observation_reports_sse_headers_before_body_eof() {
let (observer, mut receiver) = test_endpoint_observer().await;
let (mut upstream, mut upstream_peer) = tokio::io::duplex(4096);
let mut client = tokio::io::sink();
let (release_tx, release_rx) = tokio::sync::oneshot::channel();
let writer = async move {
upstream_peer
.write_all(b"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\n\r\n")
.await
.expect("write SSE headers");
let _ = release_rx.await;
};
let relay = relay_response(
"POST",
&mut upstream,
&mut client,
RelayResponseOptions {
observer: Some(&observer),
..Default::default()
},
);
let assertion = async {
let command = receiver.recv().await.expect("SSE endpoint observation");
assert!(matches!(
command,
EndpointStatusCommand::Observe {
result: EndpointResult::HttpResponseReceived,
..
}
));
let _ = release_tx.send(());
};
Box::pin(tokio::time::timeout(
std::time::Duration::from_secs(1),
async { tokio::join!(writer, relay, assertion) },
))
.await
.expect("SSE observation must not wait for body EOF")
.1
.expect("relay SSE response");
}
#[test]
fn test_parse_connection_close() {
assert!(parse_connection_close(
crates/openshell-supervisor-network/src/lib.rs
+2
View File
@@ -8,6 +8,8 @@
//! owned by the orchestrator; this crate produces denials but does not
//! aggregate them.
#[cfg(target_os = "windows")]
pub mod host;
pub mod identity;
pub mod l7;
pub mod opa;
crates/openshell-supervisor-network/src/opa.rs
+128
View File
@@ -9,9 +9,11 @@
use miette::Result;
use openshell_core::host_pattern::HostSelector;
use openshell_core::mcp::is_mcp_protocol;
use openshell_core::policy::{
FilesystemPolicy, LandlockCompatibility, LandlockPolicy, ProcessPolicy,
};
use openshell_core::policy_identity::deterministic_policy_hash;
use openshell_core::proto::SandboxPolicy as ProtoSandboxPolicy;
use openshell_policy::{L7ConfigStanza, L7Protocol as PolicyL7Protocol};
use openshell_supervisor_middleware::{ChainEntry, ChainRunner, MiddlewareRegistry};
@@ -1947,6 +1949,7 @@ fn l7_matchers_to_json(
/// kernel-resolved canonical paths reported by `/proc/<pid>/exe` (e.g.,
/// `/usr/bin/python3.11`).
fn proto_to_opa_data_json(proto: &ProtoSandboxPolicy, entrypoint_pid: u32) -> String {
let policy_hash = deterministic_policy_hash(proto);
let filesystem_policy = proto.filesystem.as_ref().map_or_else(
|| {
serde_json::json!({
@@ -2129,6 +2132,17 @@ fn proto_to_opa_data_json(proto: &ProtoSandboxPolicy, entrypoint_pid: u32) -> St
if e.provider_credentialed {
ep["provider_credentialed"] = true.into();
}
if is_mcp_protocol(&e.protocol) {
// Derive endpoint identity from the policy endpoint while
// it is still available. Request handling carries this
// opaque value through exact path selection and never
// recomputes identity from a concrete request host.
ep["endpoint_id"] =
openshell_core::endpoint_status::endpoint_id(e).into();
// The selected endpoint must retain its policy identity
// so it cannot bind to a replacement observation inventory.
ep["policy_hash"] = policy_hash.clone().into();
}
if !e.credential_signing.is_empty() {
ep["credential_signing"] = e.credential_signing.clone().into();
}
@@ -5477,6 +5491,120 @@ network_policies:
}
}
#[test]
fn l7_endpoint_config_derives_endpoint_id_from_proto() {
let mut policy = defaultable_mcp_proto(None);
let endpoint = policy
.network_policies
.get_mut("mcp")
.and_then(|rule| rule.endpoints.first_mut())
.expect("MCP test endpoint");
endpoint.path = "/mcp".to_string();
let expected = openshell_core::endpoint_status::endpoint_id(endpoint);
policy
.network_policies
.get_mut("mcp")
.expect("MCP test rule")
.binaries = vec![NetworkBinary {
path: "/usr/bin/curl".to_string(),
}];
let canonical = openshell_policy::validate_and_canonicalize_sandbox_policy(policy.clone())
.expect("canonical MCP test policy");
let expected_hash = deterministic_policy_hash(&canonical);
let engine = OpaEngine::from_proto(&policy).expect("engine from proto");
let config = engine
.query_endpoint_config(&NetworkInput {
host: "mcp.example.com".into(),
port: 443,
binary_path: PathBuf::from("/usr/bin/curl"),
binary_sha256: String::new(),
ancestors: vec![],
cmdline_paths: vec![],
})
.expect("query endpoint config")
.expect("MCP endpoint config");
let l7 = crate::l7::parse_l7_config(&config).expect("parse MCP config");
assert_eq!(
l7.endpoint_id, expected,
"OPA must carry the ID derived from the exact proto endpoint"
);
assert_eq!(l7.policy_hash, expected_hash);
}
#[tokio::test]
async fn mixed_case_mcp_policy_keeps_endpoint_observation_identity() {
use openshell_core::endpoint_status::{
EndpointConfigVersion, EndpointInventoryEntry, EndpointResult, endpoint_id,
endpoint_status_channel,
};
use openshell_core::policy_identity::deterministic_policy_hash;
for protocol in ["Mcp", "MCP"] {
let mut policy = defaultable_mcp_proto(None);
let rule = policy
.network_policies
.get_mut("mcp")
.expect("MCP test rule");
rule.endpoints[0].protocol = protocol.to_string();
rule.binaries = vec![NetworkBinary {
path: "/usr/bin/curl".to_string(),
}];
let policy = openshell_policy::validate_and_canonicalize_sandbox_policy(policy)
.expect("mixed-case MCP policy is valid");
let expected_id = endpoint_id(&policy.network_policies["mcp"].endpoints[0]);
let (sender, mut receiver) = endpoint_status_channel();
let mut tracker = receiver.tracker();
sender
.reset(
EndpointConfigVersion {
policy_hash: deterministic_policy_hash(&policy),
provider_env_revision: 1,
},
vec![EndpointInventoryEntry {
endpoint_id: expected_id.clone(),
uses_provider_credentials: false,
}],
)
.await
.expect("install mixed-case MCP inventory");
assert!(tracker.apply(receiver.recv().await.expect("inventory reset")));
let engine = OpaEngine::from_proto(&policy).expect("engine from mixed-case MCP proto");
let selected = engine
.query_endpoint_config(&NetworkInput {
host: "mcp.example.com".into(),
port: 443,
binary_path: PathBuf::from("/usr/bin/curl"),
binary_sha256: String::new(),
ancestors: vec![],
cmdline_paths: vec![],
})
.expect("query mixed-case MCP endpoint")
.expect("mixed-case MCP endpoint config");
let selected =
crate::l7::parse_l7_config(&selected).expect("parse selected MCP config");
assert_eq!(selected.protocol, crate::l7::L7Protocol::Mcp);
assert_eq!(
selected.endpoint_id, expected_id,
"{protocol} must retain inventory identity"
);
let observer = crate::l7::EndpointObserver::begin(Some(&sender), &selected)
.expect("mixed-case MCP endpoint begins an observation");
observer.observe(EndpointResult::HttpResponseReceived);
while let Ok(command) = receiver.try_recv() {
tracker.apply(command);
}
let snapshot = tracker
.snapshot()
.expect("mixed-case MCP inventory remains installed");
assert_eq!(snapshot.endpoints.len(), 1);
assert_eq!(snapshot.endpoints[0].endpoint_id, expected_id);
assert_eq!(
snapshot.endpoints[0].result,
EndpointResult::HttpResponseReceived
);
}
}
#[test]
fn l7_endpoint_config_preserves_proto_allow_encoded_slash() {
let mut network_policies = std::collections::HashMap::new();
crates/openshell-supervisor-network/src/proxy.rs
+1246 -128
View File
File diff suppressed because it is too large Load Diff
crates/openshell-supervisor-network/src/proxy/destination.rs
+38 -15
View File
@@ -9,10 +9,10 @@
//! Shared external destination validation and upstream dial boundary.
use super::{
BLOCKED_CONTROL_PLANE_PORTS, implicit_allowed_ips_for_ip_host, is_cloud_metadata_ip,
is_host_gateway_alias, is_link_local_ip, parse_allowed_ips, resolve_and_check_allowed_ips,
resolve_and_check_declared_endpoint, resolve_and_check_trusted_gateway,
resolve_and_reject_internal,
BLOCKED_CONTROL_PLANE_PORTS, DestinationCheckError, implicit_allowed_ips_for_ip_host,
is_cloud_metadata_ip, is_host_gateway_alias, is_link_local_ip, parse_allowed_ips,
resolve_and_check_allowed_ips, resolve_and_check_declared_endpoint,
resolve_and_check_trusted_gateway, resolve_and_reject_internal,
};
use ipnet::IpNet;
use openshell_core::net::{connect_tcp_nodelay_best_effort, is_always_blocked_ip, is_internal_ip};
@@ -56,6 +56,7 @@ pub(crate) struct DestinationRequest<'a> {
/// and OCSF message shapes while sharing the underlying validation logic.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum DestinationDenialKind {
Resolution,
TrustedGateway,
InvalidAllowedIps,
AllowedIps,
@@ -73,6 +74,15 @@ impl DestinationDenial {
fn new(kind: DestinationDenialKind, reason: String) -> Self {
Self { kind, reason }
}
fn from_check(error: DestinationCheckError, denied_kind: DestinationDenialKind) -> Self {
match error {
DestinationCheckError::Resolution(reason) => {
Self::new(DestinationDenialKind::Resolution, reason)
}
DestinationCheckError::Denied(reason) => Self::new(denied_kind, reason),
}
}
}
/// Select one current destination-validation mode without changing precedence.
@@ -292,37 +302,37 @@ pub(crate) async fn validate_destination(
AddressAuthorization::TrustedGatewayAlias { expected_ip } => {
resolve_and_check_trusted_gateway(host, port, *expected_ip, sandbox_entrypoint_pid)
.await
.map_err(|reason| {
DestinationDenial::new(DestinationDenialKind::TrustedGateway, reason)
.map_err(|error| {
DestinationDenial::from_check(error, DestinationDenialKind::TrustedGateway)
})?
}
AddressAuthorization::ExplicitAllowedIps(networks) => {
resolve_and_check_allowed_ips(host, port, networks, sandbox_entrypoint_pid)
.await
.map_err(|reason| {
DestinationDenial::new(DestinationDenialKind::AllowedIps, reason)
.map_err(|error| {
DestinationDenial::from_check(error, DestinationDenialKind::AllowedIps)
})?
}
AddressAuthorization::ImplicitIpLiteral(ip) => {
let network = IpNet::from(*ip);
resolve_and_check_allowed_ips(host, port, &[network], sandbox_entrypoint_pid)
.await
.map_err(|reason| {
DestinationDenial::new(DestinationDenialKind::AllowedIps, reason)
.map_err(|error| {
DestinationDenial::from_check(error, DestinationDenialKind::AllowedIps)
})?
}
AddressAuthorization::ExactDeclaredHost => {
resolve_and_check_declared_endpoint(host, port, sandbox_entrypoint_pid)
.await
.map_err(|reason| {
DestinationDenial::new(DestinationDenialKind::DeclaredEndpoint, reason)
.map_err(|error| {
DestinationDenial::from_check(error, DestinationDenialKind::DeclaredEndpoint)
})?
}
AddressAuthorization::DefaultPublicOnly => {
resolve_and_reject_internal(host, port, sandbox_entrypoint_pid)
.await
.map_err(|reason| {
DestinationDenial::new(DestinationDenialKind::InternalAddress, reason)
.map_err(|error| {
DestinationDenial::from_check(error, DestinationDenialKind::InternalAddress)
})?
}
AddressAuthorization::PinnedResolved(addresses) => addresses
@@ -375,6 +385,19 @@ mod tests {
assert_eq!(denial.kind, DestinationDenialKind::InternalAddress);
}
#[tokio::test]
async fn resolver_failure_has_a_distinct_failure_kind() {
let plan = DestinationValidationPlan {
address_authorization: AddressAuthorization::ExactDeclaredHost,
};
let denial = validate_destination(request("does-not-resolve.invalid", &plan))
.await
.err()
.expect("reserved invalid TLD must not resolve");
assert_eq!(denial.kind, DestinationDenialKind::Resolution);
}
#[tokio::test]
async fn invalid_allowed_ips_has_a_distinct_denial_kind() {
let denial = build_validation_plan(
@@ -409,7 +432,7 @@ mod tests {
expected_ip: IpAddr::V4(Ipv4Addr::LOCALHOST),
},
};
let denial = validate_destination(request("host.openshell.internal", &plan))
let denial = validate_destination(request("127.0.0.1", &plan))
.await
.err()
.expect("loopback cannot be a trusted gateway");
crates/openshell-supervisor-network/src/proxy/relay.rs
+15 -2
View File
@@ -8,6 +8,7 @@ use crate::l7::relay::L7EvalContext;
use crate::opa::{NetworkAction, OpaEngine, PolicyGenerationGuard, TunnelPolicyEngine};
use miette::{IntoDiagnostic, Result};
use openshell_core::activity::ActivitySender;
use openshell_core::endpoint_status::EndpointObservationSender;
use openshell_core::proto::ProviderProfileCredential;
use openshell_core::secrets::SecretResolver;
use std::collections::HashMap;
@@ -37,15 +38,23 @@ pub(super) struct RelayContext<'a> {
middleware_engine: &'a OpaEngine,
}
/// Non-blocking observation channels attached to an authorized HTTP relay.
pub(super) struct RelaySignals {
/// Receives general sandbox network activity.
pub(super) activity: Option<ActivitySender>,
/// Receives terminal tool server results for endpoint status reporting.
pub(super) endpoint_observation: Option<EndpointObservationSender>,
}
/// Build the request-processing context shared by CONNECT and forward HTTP.
pub(super) fn http_context(
decision: &EgressDecision,
provider_credentials: Option<openshell_core::provider_credentials::ProviderCredentialState>,
secret_resolver: Option<Arc<SecretResolver>>,
activity_tx: Option<ActivitySender>,
dynamic_credentials: Option<DynamicCredentials>,
agent_proposals: openshell_core::proposals::AgentProposals,
workspace: String,
signals: RelaySignals,
) -> L7EvalContext {
// Provider-backed credentials must be acquired from the live state for
// each request after middleware/token-grant awaits. Keep only the legacy
@@ -83,13 +92,14 @@ pub(super) fn http_context(
provider_credentials,
provider_credential_revision: None,
body_classifier: None,
activity_tx,
activity_tx: signals.activity,
dynamic_credentials: dynamic_credentials.clone(),
token_grant_resolver: dynamic_credentials
.as_ref()
.map(|_| crate::l7::token_grant_injection::default_resolver()),
agent_proposals,
workspace,
endpoint_observation_tx: signals.endpoint_observation,
}
}
@@ -360,6 +370,7 @@ mod tests {
token_grant_resolver: None,
agent_proposals: openshell_core::proposals::AgentProposals::default(),
workspace: String::new(),
endpoint_observation_tx: None,
}
}
@@ -405,6 +416,8 @@ mod tests {
configs: vec![super::super::L7ConfigSnapshot {
config: crate::l7::L7EndpointConfig {
protocol: crate::l7::L7Protocol::Rest,
endpoint_id: String::new(),
policy_hash: String::new(),
path: "/**".to_string(),
tls: crate::l7::TlsMode::Auto,
enforcement: crate::l7::EnforcementMode::Enforce,
crates/openshell-supervisor-network/src/proxy/tests/compatibility.rs
+17 -9
View File
@@ -60,6 +60,10 @@ fn assert_json_response(
#[tokio::test]
async fn destination_denials_preserve_adapter_specific_wire_contracts() {
let cases = [
(
DestinationDenialKind::Resolution,
"destination resolution failed",
),
(
DestinationDenialKind::TrustedGateway,
"trusted-gateway check failed",
@@ -381,28 +385,27 @@ network_policies:
#[cfg(not(target_os = "linux"))]
#[test]
fn identity_required_mode_is_explicitly_unsupported_off_linux() {
fn static_identity_is_supported_off_linux() {
let engine = OpaEngine::from_strings(
include_str!("../../../data/sandbox-policy.rego"),
"network_policies: {}\n",
)
.unwrap();
let binary = tempfile::NamedTempFile::new().unwrap();
std::fs::write(binary.path(), b"agent").unwrap();
let identity = ProxyIdentityMode::static_binary(binary.path()).unwrap();
let decision = authorize_egress_intent(
crate::procfs::WorkloadProxyTcpConnection::new(
"127.0.0.1:41000".parse().unwrap(),
"127.0.0.1:3000".parse().unwrap(),
),
&engine,
&BinaryIdentityCache::new(),
&AtomicU32::new(1),
&identity,
EgressIntent::connect("target.example".to_string(), 443),
);
assert!(matches!(decision.action, NetworkAction::Deny { .. }));
assert_eq!(
decision.identity,
ProcessIdentityEvidence::Unavailable(IdentityUnavailableReason::UnsupportedPlatform)
);
assert_eq!(decision.identity, ProcessIdentityEvidence::Available);
}
#[test]
@@ -541,8 +544,12 @@ network_policies:
Box::pin(handle_tcp_connection(
stream,
engine,
Arc::new(BinaryIdentityCache::new()),
Arc::new(AtomicU32::new(0)),
Arc::new(
ProxyIdentityMode::static_binary(
std::env::current_exe().unwrap(),
)
.unwrap(),
),
None,
None,
AgentProposals::default(),
@@ -553,6 +560,7 @@ network_policies:
None,
None,
None,
None,
))
.await
.unwrap();
crates/openshell-supervisor-network/src/run.rs
+30 -9
View File
@@ -26,9 +26,11 @@ use openshell_ocsf::{
use openshell_core::activity::ActivitySender;
use openshell_core::denial::DenialEvent;
use openshell_core::endpoint_status::EndpointObservationSender;
use openshell_core::proposals::AgentProposals;
use tokio::sync::mpsc::UnboundedSender;
#[cfg(target_os = "linux")]
use crate::identity::BinaryIdentityCache;
use crate::l7::tls::{
CertCache, ProxyTlsState, SandboxCa, build_upstream_client_config, read_system_ca_bundle,
@@ -36,7 +38,7 @@ use crate::l7::tls::{
};
use crate::opa::OpaEngine;
use crate::policy_local::PolicyLocalContext;
use crate::proxy::ProxyHandle;
use crate::proxy::{ProxyHandle, ProxyIdentityMode};
#[cfg(target_os = "linux")]
pub struct TransparentRuntimeSetup {
@@ -161,6 +163,13 @@ pub struct Networking {
_transparent_tcp: Option<crate::proxy::TransparentTcpHandle>,
}
#[cfg(not(target_os = "linux"))]
fn current_exe_static_identity_path() -> Result<std::path::PathBuf> {
std::env::current_exe().map_err(|e| {
miette::miette!("failed to resolve supervisor executable for static proxy identity: {e}")
})
}
/// Set up the networking stack: ephemeral CA + TLS state, proxy server,
/// and the SSH-side proxy URL / netns FD.
///
@@ -192,6 +201,7 @@ pub async fn run_networking(
openshell_endpoint: Option<&str>,
denial_tx: Option<UnboundedSender<DenialEvent>>,
activity_tx: Option<ActivitySender>,
endpoint_observation_tx: Option<EndpointObservationSender>,
agent_proposals: AgentProposals,
workspace_rx: tokio::sync::watch::Receiver<String>,
upstream_proxy_args: &crate::upstream_proxy::UpstreamProxyArgs,
@@ -308,8 +318,9 @@ pub async fn run_networking(
let _ = engine_ready_tx.send(true);
}
// Identity cache for SHA256 TOFU when OPA is active. Only consumed by
// the proxy, so it's owned here.
// Linux procfs identity mode uses a SHA256 TOFU cache. Windows host mode
// uses an explicit static sandbox identity instead.
#[cfg(target_os = "linux")]
let identity_cache = opa_engine.map(|_| Arc::new(BinaryIdentityCache::new()));
// Generate ephemeral CA and TLS state for HTTPS L7 inspection.
@@ -410,10 +421,6 @@ pub async fn run_networking(
miette::miette!("Proxy mode requires an OPA engine (--rego-policy and --rego-data)")
})?;
let cache = identity_cache.clone().ok_or_else(|| {
miette::miette!("Proxy mode requires an identity cache (OPA engine must be configured)")
})?;
// If the orchestrator gave us a proxy bind IP (the host-side veth IP
// from the workload's netns on Linux), use it so only traffic
// originating inside the namespace can reach the proxy. Otherwise the
@@ -424,12 +431,25 @@ pub async fn run_networking(
SocketAddr::new(ip, port)
});
#[cfg(target_os = "linux")]
let identity_mode = {
let cache = identity_cache.clone().ok_or_else(|| {
miette::miette!(
"Proxy mode requires an identity cache (OPA engine must be configured)"
)
})?;
ProxyIdentityMode::procfs(cache, entrypoint_pid.clone())
};
#[cfg(target_os = "windows")]
let identity_mode = ProxyIdentityMode::static_binary(current_exe_static_identity_path()?)?;
#[cfg(all(not(target_os = "linux"), not(target_os = "windows")))]
let identity_mode = ProxyIdentityMode::static_binary(current_exe_static_identity_path()?)?;
let proxy_handle = ProxyHandle::start_with_bind_addr(
proxy_policy,
bind_addr,
engine,
cache,
entrypoint_pid.clone(),
Arc::new(identity_mode),
tls_state,
Some(provider_credentials.clone()),
Some(policy_local_ctx.clone()),
@@ -437,6 +457,7 @@ pub async fn run_networking(
activity_tx.clone(),
engine_ready_rx,
upstream_proxy_args,
endpoint_observation_tx,
)
.await?;
Some(proxy_handle)
crates/openshell-supervisor-process/src/run.rs
+7 -2
View File
@@ -81,6 +81,7 @@ pub async fn run_process(
entrypoint_pid: Arc<AtomicU32>,
entrypoint_started_tx: Option<tokio::sync::oneshot::Sender<(u32, String)>>,
sidecar_exit_tx: Option<tokio::sync::mpsc::Sender<SidecarExitReport>>,
supervisor_session_updates: Option<tokio::sync::watch::Sender<Option<String>>>,
provider_credentials: ProviderCredentialState,
provider_env: std::collections::HashMap<String, String>,
ca_file_paths: Option<(std::path::PathBuf, std::path::PathBuf)>,
@@ -389,6 +390,7 @@ pub async fn run_process(
None,
Arc::clone(&supervisor_terminating),
config_apply_tx,
supervisor_session_updates,
);
info!("prepared supervisor session task resumed");
Some(task)
@@ -400,8 +402,11 @@ pub async fn run_process(
ssh_netns_fd,
None,
Arc::clone(&supervisor_terminating),
main_instance_id.clone(),
config_apply_tx,
crate::supervisor_session::SessionRuntimeContext {
instance_id: main_instance_id.clone(),
session_id_updates: supervisor_session_updates,
config_apply_tx,
},
);
info!("supervisor session task spawned");
Some(task)
crates/openshell-supervisor-process/src/supervisor_session.rs
+28 -5
View File
@@ -35,7 +35,7 @@ use openshell_ocsf::{
SeverityId, StatusId, ocsf_emit,
};
use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt};
use tokio::sync::mpsc;
use tokio::sync::{mpsc, watch};
use tokio_stream::StreamExt;
use tracing::{debug, warn};
@@ -74,6 +74,7 @@ pub struct PreparedSupervisorSession {
tx: mpsc::Sender<SupervisorMessage>,
inbound: tonic::Streaming<GatewayMessage>,
heartbeat_secs: u32,
session_id: String,
protocol_revision: u32,
bootstrap: Option<ConfigBootstrap>,
}
@@ -370,6 +371,16 @@ fn map_session_stream_message<T>(
}
}
/// Runtime identity and status channel shared with a supervisor session task.
pub struct SessionRuntimeContext {
/// Identifies the local supervisor process across gateway reconnects.
pub instance_id: String,
/// Publishes the currently accepted gateway session to sibling reporters.
pub session_id_updates: Option<watch::Sender<Option<String>>>,
/// Applies streamed configuration updates to the running supervisor.
pub config_apply_tx: Option<mpsc::Sender<ConfigApplyRequest>>,
}
/// Spawn the supervisor session task.
///
/// The task runs for the lifetime of the sandbox process, reconnecting with
@@ -382,8 +393,7 @@ pub fn spawn(
netns_fd: Option<i32>,
expected_ssh_peer_pid: Option<u32>,
terminating: Arc<AtomicBool>,
instance_id: String,
config_apply_tx: Option<mpsc::Sender<ConfigApplyRequest>>,
runtime: SessionRuntimeContext,
) -> tokio::task::JoinHandle<()> {
let config = SessionConfig {
endpoint,
@@ -392,8 +402,9 @@ pub fn spawn(
netns_fd,
expected_ssh_peer_pid,
terminating,
instance_id,
config_apply_tx,
instance_id: runtime.instance_id,
session_id_updates: runtime.session_id_updates,
config_apply_tx: runtime.config_apply_tx,
};
tokio::spawn(run_session_loop(config, None))
}
@@ -428,6 +439,7 @@ pub fn spawn_prepared(
expected_ssh_peer_pid: Option<u32>,
terminating: Arc<AtomicBool>,
config_apply_tx: mpsc::Sender<ConfigApplyRequest>,
session_id_updates: Option<watch::Sender<Option<String>>>,
) -> tokio::task::JoinHandle<()> {
let config = SessionConfig {
endpoint: prepared.endpoint.clone(),
@@ -438,6 +450,7 @@ pub fn spawn_prepared(
terminating,
instance_id: prepared.instance_id.clone(),
config_apply_tx: Some(config_apply_tx),
session_id_updates,
};
tokio::spawn(run_session_loop(config, Some((prepared, bootstrap_result))))
}
@@ -451,6 +464,9 @@ struct SessionConfig {
terminating: Arc<AtomicBool>,
instance_id: String,
config_apply_tx: Option<mpsc::Sender<ConfigApplyRequest>>,
/// Publishes the currently accepted session to sibling control-plane
/// reporters. `None` means no session is authorized to send observations.
session_id_updates: Option<watch::Sender<Option<String>>>,
}
async fn run_session_loop(
@@ -468,6 +484,9 @@ async fn run_session_loop(
} else {
run_single_session(&config).await
};
if let Some(updates) = &config.session_id_updates {
updates.send_replace(None);
}
match result {
Ok(()) => {
let event = session_closed_event(
@@ -572,6 +591,7 @@ async fn open_session(
tx,
inbound,
heartbeat_secs,
session_id: accepted.session_id,
protocol_revision,
bootstrap: (protocol_revision == SUPERVISOR_PROTOCOL_REVISION)
.then_some(accepted.bootstrap)
@@ -584,6 +604,9 @@ async fn run_prepared_session(
mut prepared: PreparedSupervisorSession,
startup_result: Option<ConfigBootstrapResult>,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
if let Some(updates) = &config.session_id_updates {
updates.send_replace(Some(prepared.session_id.clone()));
}
let heartbeat_secs = prepared.heartbeat_secs;
let channel = prepared.channel;
let tx = prepared.tx;
deploy/helm/openshell/README.md
+11 -2
View File
@@ -56,6 +56,9 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <vers
--set securityContext.runAsUser=null
```
On OpenShift 4.22+, end-to-end TLS is supported via `BackendTLSPolicy`. See the
[OpenShift install guide](https://docs.nvidia.com/openshell/latest/kubernetes/openshift#end-to-end-tls-openshift-422) for details.
## Available versions
| Tag | Source | Notes |
@@ -181,8 +184,11 @@ discovery endpoint or its TLS CA.
| certManager.enabled | bool | `false` | Create cert-manager Issuer and Certificate resources. When enabled, cert-manager owns TLS and the chart runs a JWT-only certgen hook to create the sandbox JWT signing Secret that cert-manager does not manage. |
| certManager.serverDnsNames | list | `["openshell","openshell.openshell.svc","openshell.openshell.svc.cluster.local","localhost","openshell.localhost","*.openshell.localhost","host.docker.internal"]` | DNS SANs on the cert-manager-issued server certificate. |
| certManager.serverIpAddresses | list | `["127.0.0.1"]` | IP SANs on the cert-manager-issued server certificate. |
| certManager.serverIssuerRef | object | `{"group":"","kind":"","name":""}` | Override the issuerRef for the external server Certificate (e.g. a real ACME ClusterIssuer for a publicly-trusted cert on an external hostname). When set, the chart creates a second server certificate from this issuer with only the hostnames in serverDnsNames; the internal server certificate is always signed by the chart's own CA. Leave name empty to use the chart CA for all server certificates (default). Requires certManager.enabled=true. |
| certManager.serverIssuerRef | object | `{"group":"","kind":"","name":""}` | Override the issuerRef for the external server Certificate (e.g. a real LetsEncrypt/ACME ClusterIssuer for a publicly-trusted cert on an external hostname). When set, the chart creates a second server certificate from this issuer with only the hostnames in serverDnsNames; the internal server certificate is always signed by the chart's own CA. Leave name empty to use the chart CA for all server certificates (default). Requires certManager.enabled=true. |
| fullnameOverride | string | `""` | Override the full generated resource name. |
| grpcRoute.backendTLSPolicy.caCertificateConfigMapName | string | `""` | Name of the ConfigMap containing the CA certificate (key: ca.crt) used to validate the gateway pod's TLS certificate. Defaults to `<fullname>-backend-ca` when empty. The certgen hook auto-creates this: with pkiInitJob (default), immediately on install/upgrade; with cert-manager, the hook polls for pkiInitJob.timeoutSeconds seconds waiting for cert-manager to issue the server certificate, then creates the ConfigMap. A single install usually succeeds; if cert-manager takes longer, increase pkiInitJob.timeoutSeconds. By default (pkiInitJob.failOnTimeout=true), the install fails if the timeout is reached; set failOnTimeout=false to allow the install to succeed and run `helm upgrade` after the certificate is issued. |
| grpcRoute.backendTLSPolicy.enabled | bool | `false` | Create a BackendTLSPolicy resource for end-to-end TLS between the Gateway proxy and the OpenShell gateway pod. The traffic flow is: client → HTTPS → Gateway (terminate) → TLS (re-encrypt) → gateway pod. Requires server.disableTls=false and server.tls.enableMtls=false. The certgen hook auto-creates the backend CA ConfigMap. |
| grpcRoute.backendTLSPolicy.hostname | string | `""` | Hostname the Gateway proxy validates against the backend's TLS certificate SAN. Defaults to the service FQDN (`<fullname>.<namespace>.svc.cluster.local`) when empty, which matches the SAN included by both cert-manager and the pkiInitJob. |
| grpcRoute.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. |
| grpcRoute.gateway.className | string | `"eg"` | GatewayClass to reference. Envoy Gateway installs one named "eg". |
| grpcRoute.gateway.create | bool | `false` | When true, a Gateway resource is created in the release namespace. Set to false and provide name/namespace to attach to a pre-existing Gateway. |
@@ -204,8 +210,10 @@ discovery endpoint or its TLS CA.
| openshiftRoute.enabled | bool | `false` | Create an OpenShift Route with TLS passthrough. |
| openshiftRoute.host | string | `""` | Hostname for the Route. Must match a SAN on the gateway's server cert. |
| pkiInitJob.enabled | bool | `true` | Run a pre-install/pre-upgrade Job that creates gateway and client mTLS Secrets. When certManager.enabled=true, cert-manager owns TLS and this same hook runs in JWT-only mode even if pkiInitJob.enabled remains true. |
| pkiInitJob.failOnTimeout | bool | `true` | Fail the helm install/upgrade if cert-manager does not issue the certificate within the polling timeout. When true (default), the install fails immediately if the timeout is reached, providing clear feedback that BackendTLSPolicy is non-functional. When false, the hook succeeds with a warning and you can run `helm upgrade` after cert-manager issues the certificate to create the backend CA ConfigMap. If you set this to false and see "TLS error: Secret is not supplied by SDS" when connecting to the gateway, check if the TLS secret exists and run `helm upgrade` to create the ConfigMap. |
| pkiInitJob.serverDnsNames | list | `[]` | Extra DNS SANs to append to the server certificate. |
| pkiInitJob.serverIpAddresses | list | `[]` | Extra IP SANs to append to the server certificate. |
| pkiInitJob.timeoutSeconds | int | `120` | Maximum time in seconds for the certgen hook to poll for cert-manager certificates. When using cert-manager with BackendTLSPolicy, the hook polls for this many seconds waiting for the certificate to be issued, then creates the backend CA ConfigMap. The Job deadline is set to (timeoutSeconds + 30) to allow time for ConfigMap creation and cleanup. Increase this if cert-manager takes longer than 120 seconds to issue certificates. |
| podAnnotations | object | `{}` | Extra annotations to add to the gateway pod. |
| podLabels | object | `{}` | Extra labels to add to the gateway pod. |
| podLifecycle.terminationGracePeriodSeconds | int | `5` | Grace period, in seconds, before Kubernetes terminates the gateway pod. |
@@ -286,8 +294,9 @@ discovery endpoint or its TLS CA.
| server.sandboxNamespace | string | `""` | Namespace where sandbox pods are created. Defaults to the Helm release namespace (.Release.Namespace) when left empty. |
| server.telemetryEnabled | bool | `true` | Enable anonymous OpenShell telemetry from the gateway and the sandbox supervisors it launches. |
| server.tls.certSecretName | string | `"openshell-server-tls"` | K8s secret (type kubernetes.io/tls) with tls.crt and tls.key for the server. |
| server.tls.clientCaSecretName | string | `"openshell-server-client-ca"` | K8s secret with ca.crt for client certificate verification (mTLS). Set to "" to disable mTLS and run HTTPS-only (use OIDC for auth instead). Do not set to null; omit the key to use the default secret name above. |
| server.tls.clientCaSecretName | string | `"openshell-server-client-ca"` | K8s secret with ca.crt for client certificate verification (mTLS). Only used when enableMtls is true. Set to "" to disable client certificate verification for HTTPS-only mode. |
| server.tls.clientTlsSecretName | string | `"openshell-client-tls"` | K8s secret mounted into sandbox pods for mTLS to the server. |
| server.tls.enableMtls | bool | `true` | Enable mTLS client certificate authentication. When false, the gateway runs HTTPS-only without requiring client certificates (use OIDC for auth instead). Must be false when using BackendTLSPolicy because ingress proxies cannot present client certificates to the backend. |
| server.workspaceDefaultStorageSize | string | `""` | Default storage size for the workspace PVC in sandbox pods. Uses Kubernetes quantity syntax (e.g. "2Gi", "10Gi", "500Mi"). Empty = built-in default (2Gi). |
| server.workspaceStorageClass | string | `""` | Kubernetes StorageClass for the workspace PVC in sandbox pods. Empty (default) = omit storageClassName, using the cluster's default StorageClass. Set this on clusters with no default StorageClass, otherwise the workspace PVC stays Pending and the sandbox never starts. |
| service.healthPort | int | `8081` | Gateway health service port. |
deploy/helm/openshell/README.md.gotmpl
+3
View File
@@ -56,6 +56,9 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <vers
--set securityContext.runAsUser=null
```
On OpenShift 4.22+, end-to-end TLS is supported via `BackendTLSPolicy`. See the
[OpenShift install guide](https://docs.nvidia.com/openshell/latest/kubernetes/openshift#end-to-end-tls-openshift-422) for details.
## Available versions
| Tag | Source | Notes |
deploy/helm/openshell/ci/values-openshift-route-cert-manager.yaml
+1 -1
View File
@@ -2,7 +2,7 @@
# SPDX-License-Identifier: Apache-2.0
# Render-coverage overlay for cert-manager issuing the server certificate from
# an external Issuer/ClusterIssuer (e.g. a real ACME issuer), plus an
# an external Issuer/ClusterIssuer (e.g. a real LetsEncrypt/ACME issuer), plus an
# OpenShift Route with TLS passthrough. Merge after values.yaml:
# helm lint deploy/helm/openshell -f ci/values-openshift-route-cert-manager.yaml
#
deploy/helm/openshell/templates/_helpers.tpl
+8
View File
@@ -104,6 +104,7 @@ defaults.
*/}}
{{- define "openshell.gatewayClientCaEnabled" -}}
{{- if .Values.server.disableTls -}}
{{- else if not .Values.server.tls.enableMtls -}}
{{- else if eq .Values.server.tls.clientCaSecretName "" -}}
{{- else if or .Values.server.tls.clientCaSecretName (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) -}}
true
@@ -240,6 +241,13 @@ Returns a YAML list. Append extra SANs from values with range loops.
| toYaml }}
{{- end }}
{{/*
Name of the ConfigMap holding the backend CA for BackendTLSPolicy validation.
*/}}
{{- define "openshell.backendCaConfigMapName" -}}
{{- .Values.grpcRoute.backendTLSPolicy.caCertificateConfigMapName | default (printf "%s-backend-ca" (include "openshell.fullname" .)) -}}
{{- end }}
{{/*
Gateway workload kind. StatefulSet is the default because the default SQLite
database requires persistent per-pod storage.
deploy/helm/openshell/templates/backend-tls-policy.yaml
+29
View File
@@ -0,0 +1,29 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
{{- if .Values.grpcRoute.backendTLSPolicy.enabled }}
{{- if .Values.server.disableTls }}
{{- fail "grpcRoute.backendTLSPolicy requires the gateway pod to serve TLS; set server.disableTls=false" }}
{{- end }}
{{- if .Values.server.tls.enableMtls }}
{{- fail "grpcRoute.backendTLSPolicy requires mTLS to be disabled because the Gateway proxy cannot present client certificates to the backend; set server.tls.enableMtls=false" }}
{{- end }}
apiVersion: gateway.networking.k8s.io/v1
kind: BackendTLSPolicy
metadata:
name: {{ include "openshell.fullname" . }}
namespace: {{ .Release.Namespace }}
labels:
{{- include "openshell.labels" . | nindent 4 }}
spec:
targetRefs:
- group: ""
kind: Service
name: {{ include "openshell.fullname" . }}
validation:
caCertificateRefs:
- group: ""
kind: ConfigMap
name: {{ include "openshell.backendCaConfigMapName" . }}
hostname: {{ default (printf "%s.%s.svc.cluster.local" (include "openshell.fullname" .) .Release.Namespace) .Values.grpcRoute.backendTLSPolicy.hostname }}
{{- end }}
deploy/helm/openshell/templates/certgen.yaml
+67 -1
View File
@@ -31,6 +31,11 @@ rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "create"]
{{- if .Values.grpcRoute.backendTLSPolicy.enabled }}
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "create", "update"]
{{- end }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
@@ -65,7 +70,7 @@ metadata:
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
spec:
backoffLimit: 3
activeDeadlineSeconds: 120
activeDeadlineSeconds: {{ add .Values.pkiInitJob.timeoutSeconds 30 }}
ttlSecondsAfterFinished: 300
template:
metadata:
@@ -114,4 +119,65 @@ spec:
- --server-san={{ . }}
{{- end }}
{{- end }}
{{- if and .Values.grpcRoute.backendTLSPolicy.enabled (not .Values.certManager.enabled) }}
- --backend-ca-configmap-name={{ include "openshell.backendCaConfigMapName" . }}
{{- end }}
{{- if and .Values.certManager.enabled .Values.grpcRoute.backendTLSPolicy.enabled }}
---
# Post-install Job that polls for the cert-manager-issued server Secret and
# creates / reconciles the backend CA ConfigMap. Runs after regular resources
# (including the cert-manager Certificate objects) are applied so cert-manager
# has started issuing before the poll begins.
apiVersion: batch/v1
kind: Job
metadata:
name: {{ $hookName }}-backend-ca
namespace: {{ $ns }}
labels:
{{- include "openshell.labels" . | nindent 4 }}
annotations:
helm.sh/hook: post-install,post-upgrade
helm.sh/hook-weight: "0"
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
spec:
backoffLimit: 3
activeDeadlineSeconds: {{ add .Values.pkiInitJob.timeoutSeconds 30 }}
ttlSecondsAfterFinished: 300
template:
metadata:
labels:
{{- include "openshell.selectorLabels" . | nindent 8 }}
spec:
restartPolicy: OnFailure
serviceAccountName: {{ $hookName }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: certgen
image: {{ include "openshell.image" . | quote }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
command: ["/usr/local/bin/openshell-gateway"]
args:
- generate-certs
- --jwt-only
- --jwt-secret-name={{ include "openshell.sandboxJwtSecretName" . }}
- --backend-ca-configmap-name={{ include "openshell.backendCaConfigMapName" . }}
- --backend-ca-source-secret={{ .Values.server.tls.certSecretName }}
- --backend-ca-poll-timeout-seconds={{ .Values.pkiInitJob.timeoutSeconds }}
{{- if .Values.pkiInitJob.failOnTimeout }}
- --backend-ca-fail-on-timeout
{{- end }}
{{- end }}
{{- end }}
deploy/helm/openshell/templates/gateway.yaml
+2 -2
View File
@@ -18,8 +18,8 @@ spec:
{{- if not .Values.grpcRoute.gateway.listener.tls.certificateRefs }}
{{- fail "grpcRoute.gateway.listener.tls.certificateRefs is required when grpcRoute.gateway.listener.protocol is HTTPS" }}
{{- end }}
{{- if not .Values.server.disableTls }}
{{- fail "grpcRoute.gateway.listener.protocol=HTTPS terminates TLS at Envoy Gateway, which forwards plaintext gRPC to the gateway pod; set server.disableTls=true so the pod listens plaintext (this chart does not render a BackendTLSPolicy for re-encryption to a TLS backend)" }}
{{- if and (not .Values.server.disableTls) (not .Values.grpcRoute.backendTLSPolicy.enabled) }}
{{- fail "grpcRoute.gateway.listener.protocol=HTTPS terminates TLS at the Gateway listener. Either set server.disableTls=true so the pod listens plaintext, or enable grpcRoute.backendTLSPolicy for end-to-end TLS re-encryption to the gateway pod." }}
{{- end }}
tls:
mode: Terminate
deploy/helm/openshell/tests/gateway_config_test.yaml
+18
View File
@@ -514,6 +514,24 @@ tests:
path: data["gateway.toml"]
pattern: '\[openshell\.gateway\.tls\]'
- it: omits client_ca_path from the TLS section when mTLS is disabled
template: templates/gateway-config.yaml
set:
server.tls.enableMtls: false
asserts:
- matchRegex:
path: data["gateway.toml"]
pattern: '\[openshell\.gateway\.tls\]'
- matchRegex:
path: data["gateway.toml"]
pattern: 'cert_path\s*='
- matchRegex:
path: data["gateway.toml"]
pattern: 'key_path\s*='
- notMatchRegex:
path: data["gateway.toml"]
pattern: 'client_ca_path\s*='
- it: renders client_ca_path for built-in PKI by default
template: templates/gateway-config.yaml
asserts:
deploy/helm/openshell/tests/statefulset_client_ca_test.yaml
+19
View File
@@ -64,6 +64,25 @@ tests:
- notExists:
path: spec.template.spec.volumes[3].secret.items
- it: omits client CA volume and mount when mTLS is disabled
template: templates/statefulset.yaml
set:
pkiInitJob.enabled: true
certManager.enabled: false
server.tls.enableMtls: false
asserts:
- notContains:
path: spec.template.spec.volumes
content:
name: tls-client-ca
any: true
- notContains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: tls-client-ca
mountPath: /etc/openshell-tls/client-ca
readOnly: true
# When cert-manager owns TLS, does not share its CA, and no separate client CA
# secret is configured, there is no client CA to mount: the volume must not
# render rather than mounting an empty secret name.
deploy/helm/openshell/values.yaml
+55 -3
View File
@@ -369,9 +369,14 @@ server:
tls:
# -- K8s secret (type kubernetes.io/tls) with tls.crt and tls.key for the server.
certSecretName: openshell-server-tls
# -- Enable mTLS client certificate authentication. When false, the gateway
# runs HTTPS-only without requiring client certificates (use OIDC for auth
# instead). Must be false when using BackendTLSPolicy because ingress
# proxies cannot present client certificates to the backend.
enableMtls: true
# -- K8s secret with ca.crt for client certificate verification (mTLS).
# Set to "" to disable mTLS and run HTTPS-only (use OIDC for auth instead).
# Do not set to null; omit the key to use the default secret name above.
# Only used when enableMtls is true. Set to "" to disable client certificate
# verification for HTTPS-only mode.
clientCaSecretName: openshell-server-client-ca
# -- K8s secret mounted into sandbox pods for mTLS to the server.
clientTlsSecretName: openshell-client-tls
@@ -463,6 +468,23 @@ pkiInitJob:
serverDnsNames: []
# -- Extra IP SANs to append to the server certificate.
serverIpAddresses: []
# -- Maximum time in seconds for the certgen hook to poll for cert-manager
# certificates. When using cert-manager with BackendTLSPolicy, the hook
# polls for this many seconds waiting for the certificate to be issued,
# then creates the backend CA ConfigMap. The Job deadline is set to
# (timeoutSeconds + 30) to allow time for ConfigMap creation and cleanup.
# Increase this if cert-manager takes longer than 120 seconds to issue
# certificates.
timeoutSeconds: 120
# -- Fail the helm install/upgrade if cert-manager does not issue the
# certificate within the polling timeout. When true (default), the install
# fails immediately if the timeout is reached, providing clear feedback that
# BackendTLSPolicy is non-functional. When false, the hook succeeds with a
# warning and you can run `helm upgrade` after cert-manager issues the
# certificate to create the backend CA ConfigMap. If you set this to false and
# see "TLS error: Secret is not supplied by SDS" when connecting to the gateway,
# check if the TLS secret exists and run `helm upgrade` to create the ConfigMap.
failOnTimeout: true
# cert-manager Certificate/Issuer resources (requires cert-manager CRDs in-cluster).
# Does not install cert-manager itself.
@@ -474,7 +496,7 @@ certManager:
# -- Secret created for the intermediate CA (Certificate with isCA: true).
caSecretName: openshell-ca-tls
# -- Override the issuerRef for the external server Certificate (e.g. a real
# ACME ClusterIssuer for a publicly-trusted cert on an external hostname).
# LetsEncrypt/ACME ClusterIssuer for a publicly-trusted cert on an external hostname).
# When set, the chart creates a second server certificate from this issuer
# with only the hostnames in serverDnsNames; the internal server certificate
# is always signed by the chart's own CA. Leave name empty to use the chart
@@ -548,6 +570,36 @@ grpcRoute:
# or the existing openshell-server-tls Secret (its SANs must include the
# external hostname).
certificateRefs: []
# BackendTLSPolicy for end-to-end TLS between the Gateway proxy and the
# OpenShell gateway pod. When enabled, the Gateway proxy terminates
# client-facing TLS at the listener and re-encrypts when connecting to the
# backend service, validating the backend's certificate against the
# specified CA. Requires the gateway pod to serve TLS (server.disableTls
# must be false). Supported on OpenShift 4.22+ and other platforms with
# BackendTLSPolicy support in the Gateway API implementation.
backendTLSPolicy:
# -- Create a BackendTLSPolicy resource for end-to-end TLS between the
# Gateway proxy and the OpenShell gateway pod. The traffic flow is:
# client → HTTPS → Gateway (terminate) → TLS (re-encrypt) → gateway pod.
# Requires server.disableTls=false and server.tls.enableMtls=false. The
# certgen hook auto-creates the backend CA ConfigMap.
enabled: false
# -- Name of the ConfigMap containing the CA certificate (key: ca.crt) used
# to validate the gateway pod's TLS certificate. Defaults to
# `<fullname>-backend-ca` when empty. The certgen hook auto-creates this:
# with pkiInitJob (default), immediately on install/upgrade; with
# cert-manager, the hook polls for pkiInitJob.timeoutSeconds seconds
# waiting for cert-manager to issue the server certificate, then creates the
# ConfigMap. A single install usually succeeds; if cert-manager takes longer,
# increase pkiInitJob.timeoutSeconds. By default (pkiInitJob.failOnTimeout=true),
# the install fails if the timeout is reached; set failOnTimeout=false to allow
# the install to succeed and run `helm upgrade` after the certificate is issued.
caCertificateConfigMapName: ""
# -- Hostname the Gateway proxy validates against the backend's TLS
# certificate SAN. Defaults to the service FQDN
# (`<fullname>.<namespace>.svc.cluster.local`) when empty, which matches
# the SAN included by both cert-manager and the pkiInitJob.
hostname: ""
# OpenShift Route with TLS passthrough. The gateway terminates its own
# TLS/mTLS; the router only forwards based on SNI, so it never sees plaintext
deploy/rpm/migrate-gateway-config.sh
+3 -1
View File
@@ -27,7 +27,9 @@ if [ -L "$destination" ] || { [ -e "$destination" ] && [ ! -f "$destination" ];
fi
if [ ! -e "$destination" ]; then
install -Dm 0644 "$current_default" "$destination"
destination_dir=$(dirname "$destination")
mkdir -p "$destination_dir"
install -m 0644 "$current_default" "$destination"
exit 0
fi
docs/kubernetes/ingress.mdx
+74 -3
View File
@@ -120,8 +120,9 @@ helm upgrade --install openshell \
--set grpcRoute.gateway.listener.port=443 \
--set 'grpcRoute.gateway.listener.tls.certificateRefs[0].name=openshell-ingress-tls' \
--set server.disableTls=true \
--set server.oidc.issuer=https://<issuer> \
--set 'grpcRoute.hostnames[0]=<external-hostname>'
--set server.oidc.issuer=https://keycloak.example.com/realms/openshell \
--set server.oidc.audience=openshell-cli \
--set 'grpcRoute.hostnames[0]=gateway.example.com'
```
Keep the certificate Secret in the release namespace. Referencing a Secret in another namespace requires a `ReferenceGrant`.
@@ -129,12 +130,82 @@ Keep the certificate Secret in the release namespace. Referencing a Secret in an
### Register over HTTPS
```shell
openshell gateway add https://<external-hostname> --name production --oidc-issuer https://<issuer>
openshell gateway add https://gateway.example.com \
--name production \
--oidc-issuer https://keycloak.example.com/realms/openshell \
--oidc-client-id openshell-cli
openshell status
```
See [Authentication](/kubernetes/setup) for OIDC issuer, audience, and roles configuration.
## End-to-end TLS (BackendTLSPolicy)
As an alternative to the plaintext backend path above, the chart can create a `BackendTLSPolicy` that tells the Gateway proxy to re-encrypt traffic when connecting to the OpenShell gateway pod:
```text
client → HTTPS → Gateway (terminate TLS) → TLS (re-encrypt) → openshell gateway pod
```
This keeps TLS on the gateway pod rather than disabling it with `server.disableTls=true`. The Gateway proxy validates the backend's certificate against a CA ConfigMap that the certgen hook auto-creates.
BackendTLSPolicy is a standard Gateway API resource. It is supported on OpenShift 4.22+ (via the OpenShift gateway controller) and on other platforms where the Gateway API implementation supports it (check your controller's documentation).
### Install with e2e TLS
The certgen hook automatically creates the backend CA ConfigMap when `backendTLSPolicy` is enabled:
```shell
helm upgrade --install openshell \
oci://ghcr.io/nvidia/openshell/helm-chart \
--version <version> \
--namespace openshell \
--set server.tls.enableMtls=false \
--set grpcRoute.enabled=true \
--set grpcRoute.gateway.create=true \
--set grpcRoute.gateway.className=eg \
--set grpcRoute.gateway.listener.protocol=HTTPS \
--set grpcRoute.gateway.listener.port=443 \
--set 'grpcRoute.gateway.listener.tls.certificateRefs[0].name=openshell-ingress-tls' \
--set grpcRoute.backendTLSPolicy.enabled=true \
--set server.oidc.issuer=https://keycloak.example.com/realms/openshell \
--set server.oidc.audience=openshell-cli \
--set 'grpcRoute.hostnames[0]=gateway.example.com'
```
Note that `server.disableTls` is **not** set — the gateway pod continues to serve TLS — but `server.tls.enableMtls=false` disables mTLS client certificate authentication because the Gateway proxy cannot present a client certificate to the backend. The chart will fail the install if you try to enable both `grpcRoute.backendTLSPolicy.enabled=true` and `server.tls.enableMtls=true` simultaneously. The BackendTLSPolicy hostname defaults to the service FQDN, which matches the SAN on the server certificate. Use OIDC for authentication (configured via `server.oidc.issuer`).
The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), the Certificate resources are regular release objects, and a separate post-install/post-upgrade Job (`<release>-certgen-backend-ca`) polls for up to 120 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap. This means a single `helm install` is sufficient in most cases.
If cert-manager takes longer than 120 seconds to issue certificates, increase the polling timeout with `--set pkiInitJob.timeoutSeconds=<seconds>`. The hook polls for exactly this many seconds. For example, `timeoutSeconds=180` polls for 180 seconds. By default (`pkiInitJob.failOnTimeout=true`), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional.
### Troubleshooting
If you see the error `remote connection failure, transport failure reason: TLS error: Secret is not supplied by SDS` when connecting through the Gateway:
1. Check if the backend CA ConfigMap exists:
```shell
kubectl get configmap <release-name>-backend-ca -n <namespace>
```
2. If the ConfigMap is missing, verify the TLS secret exists:
```shell
kubectl get secret <release-name>-server-tls -n <namespace>
```
3. If the secret exists but the ConfigMap doesn't, run `helm upgrade` to create it:
```shell
helm upgrade <release-name> oci://ghcr.io/nvidia/openshell/helm-chart \
--reuse-values --namespace <namespace>
```
This situation can occur if you set `pkiInitJob.failOnTimeout=false` and cert-manager issued the certificate after the hook timed out.
For OpenShift 4.22+, see [OpenShift](/kubernetes/openshift#end-to-end-tls-openshift-422) for platform-specific instructions including Gateway and GatewayClass setup.
## SSH Relay
Sandbox SSH uses the gateway endpoint registered with the CLI. No separate Helm SSH host or port values are required.
docs/kubernetes/managing-certificates.mdx
+1 -1
View File
@@ -66,7 +66,7 @@ By default, cert-manager issues both the server and client certificates from
a self-signed CA the chart creates — this rotates automatically, but the
server certificate is still not publicly trusted. `certManager.serverIssuerRef`
overrides the `issuerRef` on the server `Certificate` resource to point at a
real `Issuer` or `ClusterIssuer` instead, for example an ACME issuer:
real `Issuer` or `ClusterIssuer` instead, for example a LetsEncrypt/ACME issuer:
```shell
helm upgrade --install openshell \
docs/kubernetes/openshift.mdx
+124 -13
View File
@@ -87,13 +87,123 @@ openshell gateway add http://127.0.0.1:8080 --local --name openshift
openshell status
```
## Production: expose externally with a real certificate
## Options for end-to-end TLS
The steps above run the gateway over plaintext HTTP for quick evaluation. For
a real deployment, cert-manager can issue the gateway's server certificate
from a real Issuer or ClusterIssuer (for example, an ACME issuer), and an
OpenShift Route with TLS passthrough exposes it externally while the gateway
keeps terminating its own TLS and mTLS.
The steps above run the gateway over plaintext HTTP for quick evaluation. For production deployments, choose one of the approaches below based on your OpenShift version and preferences.
### End-to-end TLS using Gateway API and BackendTLSPolicy (OpenShift 4.22+)
OpenShift 4.22 and later support `BackendTLSPolicy` in the Gateway API, enabling end-to-end TLS between the OpenShift router and the OpenShell gateway pod. The traffic flow is:
```text
client → HTTPS → OpenShift Gateway (terminate TLS) → TLS (re-encrypt) → openshell gateway pod
```
This removes the requirement to run the gateway with `server.disableTls=true`. The OpenShift router terminates client-facing TLS at the listener and re-encrypts when connecting to the backend service, validating the backend's certificate against a CA you provide.
#### Prerequisites
- OpenShift 4.22+ cluster with the Gateway API enabled
- cert-manager installed (recommended) or the built-in pkiInitJob for server certificates
- A `GatewayClass` registered for the OpenShift gateway controller
#### Create the GatewayClass
If your cluster does not already have an OpenShift GatewayClass, create one:
```shell
oc apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: openshift-default
spec:
controllerName: openshift.io/gateway-controller/v1
EOF
```
#### Create the Gateway
Create a Gateway resource in the `openshift-ingress` namespace. Replace `<external-hostname>` with your cluster's route hostname (typically a wildcard like `*.openshell-ingress-gw.example.com`):
```shell
oc apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: openshell-gateway
namespace: openshift-ingress
spec:
gatewayClassName: openshift-default
listeners:
- name: grpc
hostname: "<external-hostname>"
port: 443
protocol: HTTPS
tls:
mode: Terminate
certificateRefs:
- name: <listener-tls-secret>
kind: Secret
allowedRoutes:
namespaces:
from: Selector
selector:
matchLabels:
kubernetes.io/metadata.name: openshell
EOF
```
The listener TLS Secret should contain the certificate for the external hostname.
#### Install with e2e TLS
Install the chart with the GRPCRoute and BackendTLSPolicy enabled. The certgen hook automatically creates the backend CA ConfigMap from the generated PKI bundle:
```shell
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
--version <version> \
--namespace openshell \
--set podSecurityContext.fsGroup=null \
--set securityContext.runAsUser=null \
--set server.tls.enableMtls=false \
--set grpcRoute.enabled=true \
--set grpcRoute.gateway.name=openshell-gateway \
--set grpcRoute.gateway.namespace=openshift-ingress \
--set 'grpcRoute.hostnames[0]=gateway.example.com' \
--set grpcRoute.backendTLSPolicy.enabled=true \
--set server.oidc.issuer=https://keycloak.example.com/realms/openshell \
--set server.oidc.audience=openshell-cli
```
| Override | Reason |
|---|---|
| `podSecurityContext.fsGroup=null` / `securityContext.runAsUser=null` | Let OpenShift's SCC admission assign UIDs. |
| `server.tls.enableMtls=false` | Disable mTLS client certificate authentication. BackendTLSPolicy only validates the server certificate; the ingress proxy cannot present a client certificate to the backend. Use OIDC for authentication instead. |
| `grpcRoute.enabled=true` | Create a GRPCRoute pointing at the external Gateway. |
| `grpcRoute.gateway.name` / `namespace` | Reference the Gateway created above in `openshift-ingress`. |
| `grpcRoute.backendTLSPolicy.enabled=true` | Create a BackendTLSPolicy for TLS re-encryption to the gateway pod. The certgen hook auto-creates the backend CA ConfigMap. The Gateway proxy validates the backend certificate against the service FQDN, which is already in the default server certificate SANs. |
| `grpcRoute.hostnames` | External hostname for the GRPCRoute. This goes on the Gateway listener certificate, not the backend certificate. |
Note that `server.disableTls` is **not** set — the gateway pod serves TLS over HTTPS without requiring client certificates. Use OIDC for authentication (see [Access Control](/kubernetes/access-control)).
**Using cert-manager instead of pkiInitJob:** Add `--set certManager.enabled=true` to the install command. The default `certManager.serverDnsNames` already includes the service FQDN needed for BackendTLSPolicy validation. The Certificate resources are regular release objects, and a separate post-install/post-upgrade Job (`<release>-certgen-backend-ca`) polls for up to 120 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap. A single `helm install` is sufficient in most cases.
If cert-manager takes longer than 120 seconds to issue certificates, increase the polling timeout with `--set pkiInitJob.timeoutSeconds=<seconds>`. The hook polls for exactly this many seconds. For example, `timeoutSeconds=180` polls for 180 seconds. By default (`pkiInitJob.failOnTimeout=true`), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional.
#### Register over HTTPS
```shell
openshell gateway add https://gateway.example.com \
--name openshift \
--oidc-issuer https://keycloak.example.com/realms/openshell \
--oidc-client-id openshell-cli
openshell status
```
### End-to-end TLS using pass-through Route (all OpenShift versions)
For OpenShift versions prior to 4.22, or when you prefer Route-based ingress, cert-manager can issue the gateway's server certificate from a real Issuer or ClusterIssuer (for example, a LetsEncrypt/ACME issuer), and an OpenShift Route with TLS passthrough exposes it externally while the gateway keeps terminating its own TLS and mTLS.
Install cert-manager and configure a working `ClusterIssuer` first — see
[Managing Certificates](/kubernetes/managing-certificates) for the
@@ -110,13 +220,13 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
--set securityContext.runAsUser=null \
--set server.disableTls=false \
--set certManager.enabled=true \
--set certManager.serverIssuerRef.name=<cluster-issuer-name> \
--set certManager.serverIssuerRef.name=letsencrypt-prod \
--set certManager.serverIssuerRef.kind=ClusterIssuer \
--set certManager.serverDnsNames[0]=<external-hostname> \
--set certManager.serverDnsNames[0]=gateway.example.com \
--set openshiftRoute.enabled=true \
--set openshiftRoute.host=<external-hostname> \
--set server.oidc.issuer=<oidc-issuer-url> \
--set server.oidc.audience=<oidc-audience>
--set openshiftRoute.host=gateway.example.com \
--set server.oidc.issuer=https://keycloak.example.com/realms/openshell \
--set server.oidc.audience=openshell-cli
```
| Override | Reason |
@@ -129,9 +239,10 @@ Register the gateway with the CLI over OIDC. Remote gateways authenticate CLI
users via OIDC, not mTLS — see [Access Control](/kubernetes/access-control):
```shell
openshell gateway add https://<external-hostname> \
openshell gateway add https://gateway.example.com \
--name openshift \
--oidc-issuer <issuer-url>
--oidc-issuer https://keycloak.example.com/realms/openshell \
--oidc-client-id openshell-cli
openshell gateway login openshift
```
docs/reference/gateway-config.mdx
+20
View File
@@ -531,6 +531,9 @@ compute_driver = "kubernetes"
[openshell.gateway.tls]
cert_path = "/etc/openshell-tls/server/tls.crt"
key_path = "/etc/openshell-tls/server/tls.key"
# client_ca_path is only rendered when server.tls.enableMtls is true (the
# default). When enableMtls is false — required for BackendTLSPolicy — the
# gateway runs HTTPS-only and this line is omitted by Helm.
client_ca_path = "/etc/openshell-tls/client-ca/ca.crt"
# When cert-manager serverIssuerRef is configured, these are populated by Helm:
# external_cert_path = "/etc/openshell-tls/server-external/tls.crt"
@@ -682,6 +685,10 @@ backend = "process_container"
default_configuration_id = "composable"
pc_least_privilege = false
pc_capabilities = []
# Pattern-C governed egress. The address is a loopback seed; the driver
# allocates a unique ephemeral port for each sandbox.
egress_proxy = false
egress_proxy_addr = ""
debug = false
etw_audit = true
```
@@ -691,6 +698,19 @@ administrator or belong to the Windows Performance Log Users group. Workload
commands and working directories remain sandbox-scoped and must be supplied in
the `mxc` driver configuration when creating a sandbox.
Set `egress_proxy = true` with a loopback seed such as
`egress_proxy_addr = "127.0.0.1:18080"` to enable the Windows Pattern-C split.
MXC redirects sandbox traffic to a sandbox-unique port on the configured
loopback IP, where the host CONNECT proxy enforces the trimmed network policy.
The driver exposes the proxy's public CA bundle through a sandbox-unique
internal filesystem grant so HTTPS clients can read the injected trust path;
the CA private key remains in gateway memory.
Governed egress requires `process_container` until MXC supports `network.proxy`
for isolation sessions.
MXC rejects policies containing `network_middlewares` before launch because
this host-proxy path does not yet receive the gateway's built-in and remote
middleware registry.
The driver records executable identity in process audit events and omits raw
command arguments because they can contain credentials or personal data. See
[OCSF JSON Export](/observability/ocsf-json-export) for durable Windows audit
docs/sandboxes/manage-sandboxes.mdx
+38 -1
View File
@@ -532,12 +532,49 @@ For automation, use `--output json` or `--output yaml` to get machine-readable s
openshell sandbox get my-sandbox --output json
```
Print only that policy YAML for scripting (same effective policy, no metadata):
Print only the policy YAML for scripting (same effective policy, no metadata):
```shell
openshell sandbox get my-sandbox --policy-only
```
### Diagnose Failed Calls to Tool Servers
A sandbox can be `Ready` while an agent's call to an external tool server fails with `fetch failed`. Run `openshell sandbox get my-sandbox` and look under `Tool server connections` for the server's address, `Last result`, and `Reported at`. This information is available for configured endpoints that use MCP over HTTP. It helps you find where the call failed without changing sandbox readiness.
JSON and YAML output include an `endpoint_statuses` list. The API and SDK expose it as `Sandbox.status.endpoint_statuses`. Each record contains `endpoint_id`, `host`, `ports`, `path`, `last_result`, and `last_reported_at`. Select the endpoint by its address to read the result directly. Treat `endpoint_id` as opaque.
For example, use `jq` to select the tool server at `tools.example.com`, path `/mcp`, and port `443`:
```shell
openshell sandbox get my-sandbox --output json | jq -e \
--arg host tools.example.com --arg path /mcp --argjson ports '[443]' '
.endpoint_statuses[]
| select(.host == $host and .path == $path and .ports == $ports)
'
```
Read `last_result` to choose the next check:
- `NoObservedExchange`: no result has been reported for the current configuration and supervisor session. Try the operation and inspect its logs if no result appears.
- `HttpResponseReceived`: the server returned a final HTTP status below 400, including a protocol upgrade. Informational responses alone do not establish success. Check the client's response for protocol or tool errors; an HTTP 200 response can still contain an error.
- `PolicyDenied`: OpenShell denied the request, including MCP protocol-version or request-body policy checks. Check the sandbox policy and denial logs.
- `CredentialUnavailable`: required credentials were unavailable. Check the endpoint's attached provider.
- `TlsFailed`: TLS setup or the handshake failed. Check certificates and TLS configuration.
- `TransportFailed`: the network exchange failed. Check name resolution, connectivity, and the server process.
- `UpstreamRejected`: the server returned an HTTP status of 400 or higher. Check the server's response and logs.
- `Unspecified`: the result is unrecognized or absent. Do not infer success from it.
For example, a stopped server can produce `TransportFailed` while the sandbox stays `Ready`. After the server recovers, an HTTP response below 400 updates the result to `HttpResponseReceived`.
Results come from actual traffic and do not expire. A result can remain unchanged after the server stops until another observed exchange or configuration/session reset. Reports combine recent observations and can drop them when full, so this list is not a request history. The address remains available when a result resets to `NoObservedExchange`. To verify current tool availability, run the actual operation.
`last_reported_at` is the RFC 3339 UTC time when the gateway accepted the result. It is empty until a result is reported. New accepted observations advance it, including repeated results; identical report retries do not. If a configuration reset supersedes a report whose acknowledgement was lost, the gateway can accept still-valid pending evidence again and advance this timestamp without another exchange.
Addresses use lowercase hosts, canonical paths, and sorted distinct effective ports. Equivalent addresses share one record that combines observations from all callers and ports; it does not establish that every caller or port works. A failure before the HTTP path is known, such as a TLS failure during CONNECT, updates status only when the host and port identify one distinct endpoint. If several configured paths share that host and port, inspect the logs for the failure.
### Inspect Logs and Activity
Stream sandbox logs to monitor agent activity and diagnose policy decisions:
```shell
e2e/parity/run.sh
+5 -3
View File
@@ -290,11 +290,13 @@ supervisor_target_triple() {
build_supervisor() {
local variant=$1 source_root=$2 target_dir=$3 override=$4 output_var=$5
local binary target jobs=()
target="$(supervisor_target_triple)"
target_dir="${target_dir:-${ROOT}/target/parity/${variant}}"
case "${target_dir}" in /*) ;; *) target_dir="${ROOT}/${target_dir}" ;; esac
binary="${override:-${target_dir}/${target}/release/openshell-sandbox}"
if [ -z "${override}" ]; then
if [ -n "${override}" ]; then
binary="${override}"
else
target="$(supervisor_target_triple)"
binary="${target_dir}/${target}/release/openshell-sandbox"
require_clean_source "${variant}" "${source_root}"
if [ -n "${CARGO_BUILD_JOBS:-}" ]; then jobs=(-j "${CARGO_BUILD_JOBS}"); fi
echo "Building ${variant} supervisor in ${target_dir}..."
e2e/parity/test.sh
+3 -1
View File
@@ -8,7 +8,9 @@
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
WORKDIR="$(mktemp -d "${TMPDIR:-/tmp}/openshell-parity-test.XXXXXX")"
TMP_ROOT="${TMPDIR:-/tmp}"
TMP_ROOT="${TMP_ROOT%/}"
WORKDIR="$(mktemp -d "${TMP_ROOT}/openshell-parity-test.XXXXXX")"
trap 'rm -rf "${WORKDIR}"' EXIT
fail() { echo "FAIL: $*" >&2; exit 1; }
e2e/rust/tests/credential_drivers.rs
-84
View File
@@ -12,17 +12,8 @@ use openshell_e2e::harness::cli::run_cli;
use openshell_e2e::harness::output::strip_ansi;
use openshell_e2e::harness::sandbox::SandboxGuard;
use sha2::{Digest, Sha256};
use tokio::io::AsyncWriteExt;
const CREDENTIAL_KEY: &str = "OPENAI_API_KEY";
const VAULT_POLICY: &str = r#"path "secret/data/openshell/provider-credentials/*" {
capabilities = ["create", "read", "update", "delete"]
}
path "secret/metadata/openshell/provider-credentials/*" {
capabilities = ["read", "delete", "list"]
}
"#;
fn unique_suffix() -> String {
let millis = SystemTime::now()
@@ -153,49 +144,6 @@ async fn bao(args: &[&str]) -> Result<String, String> {
Ok(combined)
}
async fn bao_with_stdin(args: &[&str], stdin: &str) -> Result<String, String> {
let namespace = vault_namespace();
let pod = vault_pod();
let token = vault_token();
let token_env = format!("BAO_TOKEN={token}");
let mut command = kubectl_command();
command.args([
"-n", &namespace, "exec", "-i", &pod, "--", "env", &token_env, "bao",
]);
command.args(args);
command.stdin(Stdio::piped());
command.stdout(Stdio::piped());
command.stderr(Stdio::piped());
let mut child = command
.spawn()
.map_err(|err| format!("failed to spawn bao {args:?}: {err}"))?;
let mut child_stdin = child
.stdin
.take()
.ok_or_else(|| "failed to open bao stdin".to_string())?;
child_stdin
.write_all(stdin.as_bytes())
.await
.map_err(|err| format!("failed to write bao stdin: {err}"))?;
drop(child_stdin);
let output = child
.wait_with_output()
.await
.map_err(|err| format!("failed to wait for bao {args:?}: {err}"))?;
let stdout = String::from_utf8_lossy(&output.stdout).to_string();
let stderr = String::from_utf8_lossy(&output.stderr).to_string();
let combined = format!("{stdout}{stderr}");
if !output.status.success() {
return Err(format!(
"bao {args:?} failed (exit {:?}):\n{combined}",
output.status.code()
));
}
Ok(combined)
}
async fn delete_provider(name: &str) {
let mut cmd = openshell_cmd();
cmd.arg("provider")
@@ -304,33 +252,6 @@ async fn assert_provider_placeholder_available_in_sandbox(
Ok(())
}
async fn configure_vault_storage() -> Result<(), String> {
let _ = bao(&["secrets", "enable", "-path=secret", "kv-v2"]).await;
let _ = bao(&["auth", "enable", "kubernetes"]).await;
bao(&[
"write",
"auth/kubernetes/config",
"kubernetes_host=https://kubernetes.default.svc",
"kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt",
])
.await?;
bao_with_stdin(
&["policy", "write", "openshell-provider-storage", "-"],
VAULT_POLICY,
)
.await?;
bao(&[
"write",
"auth/kubernetes/role/openshell-gateway",
"bound_service_account_names=openshell",
&format!("bound_service_account_namespaces={}", namespace()),
"policies=openshell-provider-storage",
"ttl=1h",
])
.await?;
Ok(())
}
async fn assert_kubernetes_secret_stored(
identity: &ProviderIdentity,
provider_name: &str,
@@ -457,11 +378,6 @@ async fn provider_credentials_are_stored_in_configured_backend() {
let secret_value = format!("example-e2e-{driver_slug}-{suffix}");
delete_provider(&provider_name).await;
if driver == "vault" {
configure_vault_storage()
.await
.expect("configure Vault storage fixture");
}
let result: Result<ProviderIdentity, String> = async {
create_provider(&provider_name, &secret_value).await?;
e2e/support/podman-gateway-config.sh
+5 -2
View File
@@ -128,9 +128,12 @@ e2e_write_podman_gateway_config() {
# A remote UDS driver owns all runtime options. Keep the selected
# gateway table transport-only so no in-tree setting can be mistaken
# for executed external-driver configuration.
sed -i '/^health_check_interval_secs = /d' "${output}"
sed '/^health_check_interval_secs = /d' "${output}" >"${output}.updated"
mv "${output}.updated" "${output}"
elif [ "${option_profile}" = "podman-options" ]; then
sed -i 's/^health_check_interval_secs = .*/health_check_interval_secs = 7/' "${output}"
sed 's/^health_check_interval_secs = .*/health_check_interval_secs = 7/' \
"${output}" >"${output}.updated"
mv "${output}.updated" "${output}"
fi
# The v2 template opens the Podman table. Insert gateway-owned TLS
# before it rather than reopening [openshell.gateway] later.
e2e/with-kube-gateway.sh
+68 -7
View File
@@ -202,6 +202,12 @@ cleanup_postgres_fixture() {
deploy_vault_fixture() {
echo "Deploying OpenBao fixture for Vault credential-driver validation..."
local openshift_flag="false"
if [ "${OPENSHIFT_DETECTED}" = "1" ]; then
echo "Enabling OpenBao chart OpenShift mode for restricted-v2 compatibility."
openshift_flag="true"
fi
helmctl repo add openbao https://openbao.github.io/openbao-helm \
>/dev/null 2>&1 || true
helmctl repo update openbao >/dev/null
@@ -211,6 +217,7 @@ deploy_vault_fixture() {
--set "server.dev.enabled=true" \
--set "server.dev.devRootToken=${VAULT_DEV_ROOT_TOKEN}" \
--set "injector.enabled=false" \
--set "global.openshift=${openshift_flag}" \
--wait --timeout 5m
VAULT_FIXTURE_DEPLOYED=1
@@ -219,11 +226,61 @@ deploy_vault_fixture() {
-l "app.kubernetes.io/name=openbao,component=server" \
--timeout=300s
provision_vault_auth
export OPENSHELL_E2E_VAULT_NAMESPACE="${VAULT_NAMESPACE}"
export OPENSHELL_E2E_VAULT_POD="${VAULT_RELEASE_NAME}-0"
export OPENSHELL_E2E_VAULT_TOKEN="${VAULT_DEV_ROOT_TOKEN}"
}
# Run a `bao` command in the fixture pod. Tolerates the "path is already in use"
# error from re-enabling a mount on rerun, but surfaces any other failure.
openbao_exec() {
local out
if out="$(kctl -n "${VAULT_NAMESPACE}" exec "${VAULT_RELEASE_NAME}-0" -- \
env "BAO_TOKEN=${VAULT_DEV_ROOT_TOKEN}" bao "$@" 2>&1)"; then
[ -n "${out}" ] && printf '%s\n' "${out}"
return 0
fi
case "${out}" in
*"path is already in use"*) return 0 ;;
*) printf '%s\n' "${out}" >&2; return 1 ;;
esac
}
# Provision the KV store, Kubernetes auth method, storage policy, and login role
# the gateway's Vault credential driver uses, so every provider-creating test in
# the suite can authenticate. The role binds ServiceAccount `openshell` in the
# gateway namespace, matching ci/values-credential-driver-vault.yaml.
provision_vault_auth() {
echo "Provisioning OpenBao Kubernetes auth for the gateway service account..."
openbao_exec secrets enable -path=secret kv-v2 >/dev/null
openbao_exec auth enable kubernetes >/dev/null
openbao_exec write auth/kubernetes/config \
kubernetes_host=https://kubernetes.default.svc \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
>/dev/null
printf '%s\n' \
'path "secret/data/openshell/provider-credentials/*" {' \
' capabilities = ["create", "read", "update", "delete"]' \
'}' \
'path "secret/metadata/openshell/provider-credentials/*" {' \
' capabilities = ["read", "delete", "list"]' \
'}' \
| kctl -n "${VAULT_NAMESPACE}" exec -i "${VAULT_RELEASE_NAME}-0" -- \
env "BAO_TOKEN=${VAULT_DEV_ROOT_TOKEN}" \
bao policy write openshell-provider-storage - >/dev/null
openbao_exec write auth/kubernetes/role/openshell-gateway \
bound_service_account_names=openshell \
"bound_service_account_namespaces=${NAMESPACE}" \
policies=openshell-provider-storage \
ttl=1h >/dev/null
}
cleanup_vault_fixture() {
[ -n "${KUBE_CONTEXT}" ] || return 0
[ -n "${VAULT_NAMESPACE}" ] || return 0
@@ -866,6 +923,16 @@ fi
AGENT_SANDBOX_VERSION="${AGENT_SANDBOX_VERSION}" \
bash "${ROOT}/e2e/support/install-agent-sandbox.sh" --context "${KUBE_CONTEXT}"
# Detect OpenShift up front so fixtures deployed below can apply SCC-compatible
# handling; the gateway setup further down reuses this flag.
if kctl api-resources --api-group=route.openshift.io --no-headers 2>/dev/null | grep -q .; then
OPENSHIFT_DETECTED=1
if ! command -v oc >/dev/null 2>&1; then
echo "ERROR: oc CLI is required for OpenShift SCC management but was not found." >&2
exit 2
fi
fi
ACTIVE_CREDENTIAL_DRIVER="${OPENSHELL_E2E_CREDENTIAL_DRIVER:-kubernetes-secrets}"
if [ "${OPENSHELL_E2E_CREDENTIAL_DRIVERS:-0}" = "1" ] \
&& [ "${ACTIVE_CREDENTIAL_DRIVER}" = "vault" ]; then
@@ -890,16 +957,10 @@ if [ -n "${HOST_GATEWAY_IP}" ]; then
fi
helm_values_args=(--values "${ROOT}/deploy/helm/openshell/ci/values-skaffold.yaml")
if kctl api-resources --api-group=route.openshift.io --no-headers 2>/dev/null | grep -q .; then
OPENSHIFT_DETECTED=1
if [ "${OPENSHIFT_DETECTED}" = "1" ]; then
echo "OpenShift detected — applying SCC-compatible security context overrides."
helm_values_args+=(--values "${ROOT}/deploy/helm/openshell/ci/values-openshift-scc.yaml")
if ! command -v oc >/dev/null 2>&1; then
echo "ERROR: oc CLI is required for OpenShift SCC management but was not found." >&2
exit 2
fi
kctl create namespace "${NAMESPACE}" --dry-run=client -o yaml | kctl apply -f -
echo "Granting privileged SCC to openshell-sandbox in namespace ${NAMESPACE}..."
examples/governance-interceptor/Cargo.lock
Generated
+1
View File
@@ -1075,6 +1075,7 @@ dependencies = [
"rustls-pemfile",
"serde",
"serde_json",
"sha2",
"thiserror",
"tokio",
"tokio-stream",
examples/supervisor-middleware-content-guard/Cargo.lock
Generated
+72
View File
@@ -201,6 +201,15 @@ version = "2.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8"
[[package]]
name = "block-buffer"
version = "0.10.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71"
dependencies = [
"generic-array",
]
[[package]]
name = "bytes"
version = "1.12.0"
@@ -302,6 +311,35 @@ version = "0.8.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
[[package]]
name = "cpufeatures"
version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280"
dependencies = [
"libc",
]
[[package]]
name = "crypto-common"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
dependencies = [
"generic-array",
"typenum",
]
[[package]]
name = "digest"
version = "0.10.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
dependencies = [
"block-buffer",
"crypto-common",
]
[[package]]
name = "displaydoc"
version = "0.2.6"
@@ -425,6 +463,16 @@ dependencies = [
"slab",
]
[[package]]
name = "generic-array"
version = "0.14.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
dependencies = [
"typenum",
"version_check",
]
[[package]]
name = "getrandom"
version = "0.2.17"
@@ -914,6 +962,7 @@ dependencies = [
"rustls-pemfile",
"serde",
"serde_json",
"sha2",
"thiserror",
"tokio",
"tokio-stream",
@@ -1421,6 +1470,17 @@ dependencies = [
"zmij",
]
[[package]]
name = "sha2"
version = "0.10.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"
dependencies = [
"cfg-if",
"cpufeatures",
"digest",
]
[[package]]
name = "shlex"
version = "2.0.1"
@@ -1789,6 +1849,12 @@ version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b"
[[package]]
name = "typenum"
version = "1.20.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20"
[[package]]
name = "unicase"
version = "2.9.0"
@@ -1849,6 +1915,12 @@ version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
[[package]]
name = "version_check"
version = "0.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
[[package]]
name = "want"
version = "0.3.1"

Some files were not shown because too many files have changed in this diff Show More